RU2780422C1 - Method and apparatus for video encoding - Google Patents

Abstract

FIELD: encoding.

SUBSTANCE: invention relates to video encoding tools. The encoded information of the coding block (CB) is decoded in an image from the bitstream of the encoded video, wherein the encoded information indicates the width of the CB equal to W counts and the height of the CB equal to H counts. The CB is divided into sub-processing units (SPU) with a width and height, wherein the width of a SPU is equal to the minimum value from W and K, or the height of a SPU is equal to the minimum value from H and K. At least one of the width W and height H for the CB herein is greater than the size K of the processing data unit. The partitioning structure for the additional partitioning of the SPU is determined based on the width and height of the SPU and the maximum size of the transformation unit (TU), constituting M samples, wherein at least one from the width and height of the SPU is greater than M. Each of the SPU is divided into TUs with the dimensions M×M based on the determined partitioning structure.

EFFECT: increase in the efficiency of video encoding.

14 cl, 58 dwg

Description

INCLUSION BY LINK

[0001] The present invention claims priority of U.S. Patent Application No. 16/823,831, "Method and Apparatus for Video Coding", filed March 19, 2020, which claims priority of U.S. Provisional Application No. 62/822,787, "Modified VPDU Compatible Max Transform Control" filed March 22, 2019. The contents of previous applications are hereby incorporated by reference in their entirety.

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

[0002] This application describes embodiments generally related to video coding.

BACKGROUND OF THE INVENTION

[0003] The description of the prior art is provided herein for the purpose of presenting the overall context of the invention. The work of the authors of the invention, to the extent that it is described in this section, as well as aspects of the description that cannot be qualified as state of the art at the time of filing, neither expressly nor impliedly recognized as the state of the art underlying the present invention.

[0004] Encoding and decoding of the video signal may be performed using inter-picture prediction with motion compensation. Uncompressed digital video may include a sequence of images, each of which has a spatial size of, for example, 1920×1080 luma samples and their associated chrominance samples. The image sequence may have a fixed or variable image rate (informally also referred to as frame rate), such as 60 images per second, or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (resolution of 1920x1080 luma samples at 60Hz frame rate) requires about 1.5Gbps of bandwidth. An hour of such video requires more than 600 GB of storage.

[0005] One goal of video encoding and decoding may be to reduce redundancy in the input video signal by compressing. Compression can help to alleviate the aforementioned bandwidth or storage requirements, in some cases by two orders of magnitude or more. You can use both lossless and lossy compression, as well as a combination of both. Lossless compression refers to techniques for reconstructing an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the discrepancy between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. Lossy compression is widely used for video. The amount of distortion allowed depends on the application; for example, users of some custom streaming applications can tolerate higher distortion than users of broadcast TV applications. The achievable compression ratio may reflect that a higher allowed/allowable distortion may result in a higher compression ratio.

[0006] The video encoder and decoder may use techniques from several broad categories including, for example, motion compensation, transformation, quantization, and entropy coding.

[0007] Video codec technologies may include techniques known as intra-coding. With internal (intra-) coding, sample values are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, the image is spatially divided into blocks of samples. When all blocks of samples are encoded in intra-mode, the picture is called an intra-picture. Intra-pictures and their derivatives, such as pictures obtained in the decoder independent update mode, can be used to reset the decoder state, allowing them to be used as the first picture in the encoded video bitstream and video session, or as a still picture. The intra-block samples may be transformed and the transform coefficients may be quantized prior to entropy coding. Intra-prediction may be a technique for minimizing sample values in the pre-transform region. In some cases, the smaller the value of DC (constant component) after the transformation, and the smaller the AC (variable components) coefficients, the fewer bits are required for a given quantization step size to represent a block after entropy coding.

[0008] Conventional intra coding known, for example, from MPEG-2 generation coding technologies, does not use intra prediction. However, some more recent video compression techniques include techniques that attempt, for example, from ambient sample data and/or metadata obtained during encoding/decoding of spatially adjacent and preceding data blocks in decoding order. Such methods are hereinafter referred to as "internal prediction" methods. Note that, in at least some cases, intra prediction is only performed using reference data from the current picture to be reconstructed, and not from reference pictures.

[0009] There may be many different forms of intra prediction. When more than one of these methods can be used in a given video coding technology, the method used can be encoded in intra prediction mode. In some cases, modes may have sub-modes and/or parameters that may be encoded separately or included in the mode codeword. The use of a particular codeword for a given combination of modes/submodes/parameters can affect the gain in coding efficiency through intra prediction and hence the entropy coding technology used to translate the codewords into the bitstream.

[0010] A certain mode of intra prediction was first specified in H.264, refined in H.265, and further refined in more modern coding technologies, for example, the joint exploration model (JEM, joint exploration model), universal video coding (VVC, versatile video coding) and a set of standards (BMS, benchmark set). The predictor block can be formed using the values of neighboring samples belonging to already available samples. The values of neighboring samples are copied into the predictor block according to the direction. The reference to the direction to be used may be encoded in the bitstream, or may itself be predicated.

[0011] In the lower right corner of FIG. 1A shows a subset of nine set predictor directions known from 33 possible H.265 predictor directions (corresponding to 33 angle modes out of 35 intra modes). The point (101) where the arrows converge represents the predicted reading. The arrows represent the direction from which the reading is predicted. 45 degrees to the horizontal. Likewise, arrow (103) indicates that sample (101) is predicted from the sample(s) located down to the left of sample (101), at an angle of 22.5 degrees to the horizontal.

[0012] In the upper left corner of FIG. 1A depicts a square block (104) of 4×4 samples (indicated by the heavy dotted line). The square block (104) includes 16 samples, each labeled with an "S", a y-axis position (eg, row index) and an x-axis position (eg, column index). For example, sample S21 is the second sample on the Y axis (top) and the first sample (left) on the X axis. Similarly, sample S44 is the fourth sample in block (104) on both the Y and X axes. Since the block has a size of 4×4 samples , S44 is located at the bottom right. Additionally, reference readings are shown, which follow a similar numbering scheme. The reference sample is denoted by the letter R, its Y-position (eg, row index) and X-position (column index) relative to the block (104). In both H.264 and H.265, the predicted samples are adjacent to the block to be reconstructed; so there is no need to use negative values.

[0013] Intra-picture prediction may work by copying sample reference values from neighboring samples indicated by the signaled prediction direction. For example, suppose the coded video bitstream includes signaling which, for this block, indicates the direction of prediction indicated by the arrow (102) - that is, the samples are predicted from the predicted sample(s) located at the top. on the right, at an angle of 45 degrees to the horizontal. In this case, samples S41, S32, S23 and S14 are predicted from the same reference sample R05. Then, the sample S44 is predicted from the reference sample R08.

[0014] In some cases, the values of a plurality of reference samples may be combined, such as by interpolation, to calculate a reference sample; especially when the directions are not multiples of 45 degrees.

[0015] The number of possible directions increases as video coding technologies develop. In H.264 (2003), nine different directions could be represented. In H.265 this has increased to 33 (2013), and JEM/VVC/BMS, at the time of writing this description, can support up to 65 destinations. Experiments have been made to identify the most likely directions, and some entropy encoding techniques are used to represent these likely directions with a small number of bits, at the cost of losing some of the less likely directions. Additionally, the directions themselves can sometimes be predicted from adjacent directions used in adjacent already decoded blocks.

[0016] FIG. 1B is a diagram (180) showing 65 intra prediction directions according to JEM to illustrate an increase in the number of prediction directions over time.

[0017] The mapping of intra prediction direction bits in the encoded video bitstream that represent the direction may differ from one video coding technology to another video coding technology; and may cover, for example, from simple forward prediction direction mapping to intra prediction mode, code words, complex adaptive schemes involving most likely modes, and similar techniques. However, in all cases, there may be some trends that are statistically less likely to occur in video content than for some other trends. Because the goal of video compression is to reduce redundancy, these less likely directions will, in a well-performing video coding technology, be represented by more bits than more likely directions.

[0018] Motion compensation may be a lossy compression method and may refer to methods where a block of sample data from a previously reconstructed image or part of it (reference image) is spatially shifted in the direction indicated by the motion vector (hereinafter MV, motion vector) , is used to predict a newly reconstructed image or part of an image. In some cases, the reference image may be identical to the image currently being reconstructed. MVs may have two X and Y dimensions, or three dimensions, with the third indicating the reference picture used (the latter, indirectly, may be a temporal dimension).

[0019] In some video compression techniques, the MV applicable to a certain region of sample data may be predicted from other MVs, such as those associated with another region of sample data, spatially adjacent to the reconstructed region, and preceding that MV in decoding order. This can significantly reduce the amount of data required for MV encoding, thereby eliminating redundancy and increasing the compression ratio. MV prediction can work efficiently, for example, due to the fact that when encoding the input video signal received from the camera (known as natural video), there is a statistical probability that areas larger than the area to which a single MV is applied move in similar direction, and therefore in some cases can be predicted using a similar motion vector obtained from the MV of the neighboring area. As a result, the MV found for a given region is similar or identical to the MV predicted from the surrounding MVs, which in turn can be represented, after entropy encoding, in fewer bits than would be used when encoding the MV directly. In some cases, MV prediction can serve as an example of lossless compression of a signal (namely, MV) obtained from the original signal (namely, a stream of samples). In other cases, the MV prediction itself may be lossy, for example, due to rounding errors when computing a predictor from multiple surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265/HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Of the many MV prediction mechanisms offered in H.265, a technique hereinafter referred to as "spatial pooling" is described.

[0021] Referring to FIG. 2, the current block (201) contains the samples found by the encoder during the motion search process to predict from a previous block of the same size that is spatially shifted. Instead of encoding this MV directly, the MV can be obtained from the metadata associated with one or more reference pictures, eg, from the most recent (in decoding order) reference picture, using the MV associated with any of the five surrounding samples, labeled A0, A1, and B0, B1, B2 (202-206, respectively). In H.265, MV prediction may use predictors from the same reference picture as the neighboring block uses.

SUMMARY OF THE INVENTION

[0022] Aspects of the invention provide methods and apparatus for encoding/decoding a video signal. In some examples, the video decoding apparatus includes a processing circuit. The processing circuit is configured to decode encoded coding block information (CB) in an image from an encoded video bitstream. The encoded information indicates the width W of the samples and the height H of the samples CB. The processing scheme may partition the CB into sub-processing units (SPUs) having a width equal to the minimum value of W and K and a height equal to the minimum value of H and K. At least one of the width W and the height H for CB is greater than the size K of the processing data unit. The processing circuit may determine the split structure for the additional split of the SPU based on the width and height of the SPU and the maximum size of the transform unit (TU) M samples. At least one of the width and height of the SPUs is greater than M. The processing circuit may partition each of the SPUs into TUs of size MxM based on the determined partitioning structure.

[0023] According to an embodiment, the width and height of the SPU is greater than M. The processing circuit may determine the partition structure as a quadtree partition structure. The processing scheme may partition SPUs into TUs based on a quadtree partition structure.

[0024] According to an embodiment, the width of the SPU is greater than M and the height of the SPU is M. The processing circuit may determine the partition structure as a vertical partition structure in the form of a binary tree. The processing circuit may partition SPUs into TUs based on a binary tree vertical partitioning structure.

[0025] According to an embodiment, the height of the SPU is greater than M and the width of the SPU is M. The processing circuit may determine the split structure as a horizontal split structure in the form of a binary tree. The processing circuit may partition SPUs into TUs based on a horizontal binary tree partitioning structure.

[0026] According to an embodiment, the processing scheme may recursively partition one of the SPUs into TUs based on the partition structure.

[0027] According to an embodiment, the processing circuit may process the SPUs according to the first scan order, and process the TUs in each of the SPUs according to the second scan order. In an example, at least one of the first scan order and the second scan order is one of (i) raster scan order, (ii) vertical scan order, (iii) zigzag scan order, and (iv) diagonal scan order. In the example, the first scan order and the second scan order are raster. In the example, W is 128, H is 64, K is 64, and M is 32. The first scan order is left-to-right order, and the second scan order is raster scan order.

[0028] According to an embodiment, the processing data unit size K indicates the virtual pipeline data unit (VPDU) size. The first of the SPUs is included in the first VPDU and the second of the SPUs is included in the second VPDU in the picture. After processing the first VPDU in the first stage of the multi-stage pipeline, the processing circuit may simultaneously process the first VPDU in the second stage of the multi-stage pipeline and the second VPDU in the first stage.

BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Additional features, nature and various advantages of the disclosed invention will be apparent from the following detailed description and the accompanying drawings, in which:

[0030] FIG. 1A is a diagram of an exemplary subset of intra prediction modes;

[0031] FIG. 1B is a diagram of intra prediction directions;

[0032] FIG. 2 is a diagram of the current block and its surrounding space merge candidates in one example;

[0033] FIG. 3 is a simplified block diagram of a communications system (300) in accordance with an embodiment;

[0034] FIG. 4 is a simplified block diagram of a communication system (400) in accordance with an embodiment;

[0035] FIG. 5 is a simplified block diagram of a decoder according to an embodiment;

[0036] FIG. 6 is a simplified block diagram of an encoder according to an embodiment;

[0037] FIG. 7 is a block diagram of an encoder according to another embodiment;

[0038] FIG. 8 is a block diagram of a decoder according to another embodiment;

[0039] FIG. 9A shows a CTU that is partitioned with a quadtree plus binary tree (910) structure (QTBT);

[0040] FIG. 9 V - structure (920) QTBT;

[0041] FIG. 9C - horizontal center-side ternary tree;

[0042] FIG. 9D - vertical center-side ternary tree;

[0043] FIG. 10A-10D show 4-point, 8-point, 16-point, and 32-point DCT-2 transform kernel matrices, respectively;

[0044] FIG. 11A-11E is a 64x64 matrix of the transform kernel of the 64-point DCT-2 transform;

[0045] FIG. 12 - basic transformation functions of the selected discrete sine transform (DST, discrete sine transform) / discrete cosine transform (DCT, discrete cosine transform) adaptive multiple transform (AMT, adaptive multiple transform);

[0046] FIG. 13 is a table (1300) illustrating a mapping relationship between an mts_idx value and corresponding horizontal or vertical transformations;

[0047] FIG. 14A-14D show DST-7 transform transform kernel matrices;

[0048] FIG. 15A-15D show DCT-8 transform transformation kernel matrices;

[0049] FIG. 16 - the number of sub-parts depending on the size of the block;

[0050] FIG. 17 - an example of intra-sub-partition (ISP, intra sub-partition);

[0051] FIG. 18 is an example of an ISP.

[0052] FIG. 19A-19B show example syntax elements (1900) for ISP encoding mode;

[0053] FIG. 20A-20D show examples of sub-block transform (SBT, sub-block transform);

[0054] FIG. 21A-21I show an example of a textual specification of a video coding standard when using SBT;

[0055] FIG. 22 shows various YUV formats used in some embodiments;

[0056] FIG. 23 - examples of forbidden partitioning in the form of a ternary tree (ТТ, ternary tree) and a binary tree (BT, binary tree);

[0057] FIG. 24 is an example of a transformation tree syntax;

[0058] FIG. 25 - block (2510) encoding size 128×64 samples;

[0059] FIG. 26A - block (2610A) encoding size 128×32 samples;

[0060] FIG. 26B - coding block (2610B) with a size of 128×32 samples;

[0061] FIG. 27 is a block diagram of the operations of a process (2700) according to an embodiment of the invention;

[0062] FIG. 28 is a diagram of a computer system in accordance with an embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

[0063] I. Encoder and decoder for video coding

[0064] FIG. 3 shows a simplified block diagram of a communication system (300) according to an embodiment of the present invention. The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminals (310) and (320) interconnected via a network (350). In the example shown in FIG. 3, the first pair of terminals (310) and (320) performs a unidirectional data transfer. For example, terminal device (310) may encode video data (eg, a video stream captured by terminal device (310)), for transmission to another terminal device (320) via network (350). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to recover the video images, and display the video images according to the recovered video data. One-way communication may be inherent in media service applications and the like.

[0065] In another example, the communication system (300) includes a second pair of terminals (330) and (340) that carry out bi-directional transmission of encoded video data that may occur, for example, during a video conference. For bidirectional data communication, by way of example, each terminal of terminals (330) and (340) may encode video data (eg, a video stream captured by the terminal) for transmission to the other terminal of terminals (330) and (340). ) via the network (350). Each terminal of terminals (330) and (340) may also receive encoded video data transmitted by another terminal from terminals (330) and (340) and may decode the encoded video data to reconstruct video images and display video images on an available display device according to recovered video data.

[0066] In the example shown in FIG. 3, terminal devices (310), (320), (330), and (340) can be illustrated as servers, personal computers, and smartphones, but this does not limit the principles of the present invention. Embodiments of the present invention find application in laptop computers, tablet computers, media players and/or dedicated video conferencing equipment. Network (350) represents any number of networks that carry encoded video data between terminals (310), (320), (330) and (340), including, for example, wired and/or wireless communication networks. The communication network (350) allows data to be exchanged in a circuit-switched and/or packet-switched mode. Exemplary networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may not be relevant to the present invention unless specifically noted below.

[0067] FIG. 4 shows, as an example of the application of the disclosed invention, the placement of a video encoder and a video decoder in a streaming environment. The disclosed invention may be equally applicable to other video processing applications, including, for example, video conferencing, digital television, storage of compressed video on digital media, including CD, DVD, memory card, and the like.

[0068] The streaming system may include a capture subsystem (413), which may include a video source (401), such as a digital camera, producing, for example, an uncompressed video stream (402). For example, the video stream (402) includes samples taken by a digital camera. The video stream (402), shown in bold to emphasize the large amount of data compared to the encoded video data (404) (or encoded video bitstreams), may be processed by an electronic device (420) that includes a video encoder (403) connected to source (401) of the video signal. The video encoder (403) may include hardware, software, or a combination thereof for providing or implementing aspects of the disclosed invention, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted with a thin line to emphasize the smaller amount of data compared to the video stream (402), may be stored on the streaming server (405) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) in FIG. 4 may access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include a video decoder (410), such as in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and creates an outgoing video stream (411) that can be rendered on a display (412) (eg, a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (eg, video bitstreams) may be encoded according to one or another video coding/compression standard. Examples of these standards include ITU-T Recommendation H.265. For example, a video coding standard is being developed, officially called Universal Video Coding (VVC, Universal Video Coding). The disclosed invention may be used in the context of VVC.

[0069] Note that the electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0070] FIG. 5 shows a block diagram of a video decoder (510) according to an embodiment of the present invention. The video decoder (510) may be included in the electronic device (530). The electronic device (530) may include a receiver (531) (eg, receiver circuits). Video decoder (510) may be used instead of video decoder (410) in the example shown in FIG. four.

[0071] The receiver (531) may receive one or more encoded video sequences for decoding by the video decoder (510); in the same or another embodiment, one coded video sequence at a time, where the decoding of each coded video sequence is independent of other coded video sequences. The encoded video sequence may be received from a channel (501), which may be a hardware/software link to a storage device where the encoded video data is stored. The receiver (531) may receive encoded video data with other data, such as encoded audio data and/or auxiliary data streams, which may be relayed to appropriate users and entities (not shown). The receiver (531) may separate the encoded video sequence from other data. To combat network jitter, a buffer memory (515) may be connected between the receiver (531) and the entropy decoder/analyzer (520) (hereinafter "analyzer (520)"). In some applications, the buffer memory (515) is included in the video decoder (510). In others, it may not be part of the video decoder (510) (not shown). In other embodiments, there may be a buffer memory (not shown) outside the video decoder (510), for example, to combat network jitter, in addition to another buffer memory (515) within the video decoder (510), for example, to control the timing of the playout. When the receiver (531) receives data from a storage/relay device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may not be needed or may be small. For use in best attempt packet networks, such as the Internet, buffer memory (515) may be required, may be relatively large, and may have a predominantly adaptive size, and may at least partially be implemented in an operating system or similar elements (not shown) outside video decoder (510).

[0072] The video decoder (510) may include an analyzer (520) for reconstructing the symbols (521) from the encoded video sequence. The categories of these symbols include information used to control the operation of the video decoder (510), and possibly information to control a visualization device, such as a visualization device (512) (for example, a display screen), which is not an integral part of the electronic device (530), but may be connected to an electronic device (530) as shown in FIG. 5. The control information for the rendering device(s) may be Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The analyzer (520) may analyze/entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be in accordance with a video coding technology or standard, and may follow various principles, including variable run length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so on. The analyzer (520) can extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOP, Groups of Pictures), images, tiles, slices, macroblocks, coding units (CU, Coding Units), blocks, transformation units (TU, Transform Units), prediction units (PU, Prediction Units ) etc. The analyzer (520) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0073] The analyzer (520) may perform an entropy decoding/analysis operation of the video sequence received from the buffer memory (515) to create symbols (521).

[0074] Several different modules may be used for symbol reconstruction (521) depending on the type of encoded video image or parts thereof (eg: inter- and intra-pictures, inter- and intra-block) and other factors. Which modules are used, and how, may be determined by the subgroup management information extracted from the encoded video sequence by the analyzer (520). The flow of such subgroup management information between the analyzer (520) and multiple modules is not shown hereinafter for simplicity.

[0075] In addition to the previously mentioned functional blocks, the video decoder (510) can be fundamentally divided into several functional modules, as described below. In a practical implementation operating under commercial constraints, many of these modules interact closely with one another and can, at least partially, be nested within one another. However, for the purposes of describing the disclosed invention, a fundamental division into the following functional modules is appropriate.

[0076] The first module is the scaling/inverse transform module (551). The scaling/inverse transform module (551) receives the quantized transform coefficient as well as control information including the transform used, block size, quantization factor, quantization scaling matrices, and so on. as character(s) (521) from the analyzer (520). The scaling/inverse transform module (551) can output blocks containing sample values that can be input to the aggregator (555).

[0077] In some cases, the output samples of the scaling / inverse transform block (551) may refer to an intra-coded block; that is, a block that does not use predicted information from previously reconstructed images, but may use predicted information from previously reconstructed portions of the current image. Such predicted information may be provided by an intra-picture prediction module (552). In some cases, the intra-image prediction module (552) generates a block of the same size and shape as the block to be reconstructed using the previously reconstructed environment information retrieved from the current image buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed current picture and/or a fully reconstructed current picture. The aggregator (555) sometimes adds, for each sample, the prediction information generated by the intra prediction module (552) to the output sample information provided by the scaling/inverse transform module (551).

[0078] In other cases, the output samples of the scaling/inverse transform module (551) may refer to an intra-coded block, possibly with motion compensation. In such a case, the motion compensation prediction module (553) may access the reference picture memory (557) to retrieve samples used for prediction. After applying motion compensation to the extracted samples according to the symbols (521) associated with the block, these samples can be added by the aggregator (555) to the output signal of the scaling / inverse transform module (551) (in this case referred to as residual samples or residual signal) for generation of output sample information. The addresses in the reference picture memory (557) from where the motion compensation prediction module (553) retrieves the predicted samples may be governed by the motion vectors available to the motion compensation prediction module (553) in the form of symbols (521), which may have, for example, X, Y components and reference image. Motion compensation may also include interpolation of sample values retrieved from reference picture memory (557) when accurate sub-sample motion vectors, motion vector prediction mechanisms, etc. are used.

[0079] Various loop filtering methods can be applied to the output samples of the aggregator (555) in the loop filter module (556). The video compression technologies may include deblocking filter technologies driven by parameters included in the encoded video sequence (also referred to as the encoded video bitstream) and made available to the loop filter module (556) as symbols (521) from the analyzer (520), but may also respond to meta-information obtained during the decoding of the previous (in the order of decoding) parts of the encoded image or encoded video sequence, as well as respond to previously reconstructed and loop-filtered sample values.

[0080] The loop filter module (556) may provide a stream of samples that may be fed to the renderer (512) as well as stored in the reference picture memory (557) for use in future inter-picture prediction.

[0081] Some encoded pictures, when fully reconstructed, can be used as reference pictures for future prediction. For example, when the encoded picture corresponding to the current picture is completely reconstructed and the encoded picture is identified as a reference picture (e.g., by the analyzer (520)), the current picture buffer (558) may become part of the reference picture memory (557), and a fresh current picture buffer may be re-allocated before the reconstruction of the next encoded picture starts.

[0082] The video decoder (510) may perform decoding operations according to a predetermined video compression technology, such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used, in the sense that the encoded video sequence may adhere to both the syntax of the video compression technology or standard and the profiles documented in the video compression technology or standard. In particular, a profile may select certain tools as tools available for use only under that profile, from among all the tools available in a video compression technology or standard. It may also be necessary for consistency that the complexity of the encoded video sequence remain within the boundaries set by the video compression technology level or standard. In some cases, the levels limit the maximum image size, the maximum frame rate, the maximum sampling rate for reconstruction (measured in megasamples per second, for example), the maximum reference image size, and so on. The limits set by the levels, in some cases, may be further limited by the specifications of the hypothetical reference decoder (HRD, Hypothetical Reference Decoder) and metadata for managing the HRD buffer signaled in the encoded video sequence.

[0083] According to an embodiment, the receiver (531) may receive additional (redundant) data with encoded video. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (510) to correctly decode the data and/or more accurately reconstruct the original video data. The additional data may be, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, and so on.

[0084] FIG. 6 is a block diagram of a video encoder (603) according to an embodiment of the present invention. The video encoder (603) is included in the electronic device (620). The electronic device (620) includes a transmitter (640) (eg, a transmission circuit). Video encoder (603) may be used instead of video encoder (403) in the example shown in FIG. four.

[0085] The video encoder (603) may receive video samples from a video source (601) (which is not part of the electronic device (620) in the example shown in FIG. 6) that may capture the video image(s) to be encoding with a video encoder (603). In another example, the video signal source (601) is included in the electronic device (620).

[0086] The video signal source (601) may provide the source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream of any suitable bit depth (eg: 8 bit, 10 bit, 12 bit, ...), any color space (eg, BT.601 Y CrCB, RGB, ...) and any suitable sampling structure (eg Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (601) may be a storage device where previously prepared video is stored. In a videoconferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual images that give a sense of movement when viewed in sequence. The images themselves may be organized as a spatial array of pixels, where each pixel may contain one or more samples depending on the sampling structure, color space, etc. used. It is not difficult for a person skilled in the art to understand the relationship between pixels and samples. The following description is for samples.

[0087] According to an embodiment, the video encoder (603) may encode and compress images of the original video sequence into the encoded video sequence (643) in real time or subject to any other time constraints imposed by the application. Establishing a proper coding rate is one of the functions of the controller (650). In some embodiments, the controller (650) controls other functional modules, as described below, and is operatively connected to other functional modules. The connection is not shown for simplicity. The parameters set by the controller (650) may include parameters related to frequency adjustment (picture skip, quantizer, lambda value applied in rate-distortion optimization, ...), picture size, group of pictures (GOP) scheme, maximum vector search area movements, etc. The controller (650) may be configured to have other suitable functions related to the video encoder (603) optimized for a given system design.

[0088] In some embodiments, video encoder (603) is configured to operate in a coding loop. As a very simplified description, by way of example, an encoding loop may include a source encoder (630) (e.g., responsible for generating symbols, e.g., a stream of symbols, based on the input image to be encoded and the reference image(s) )) and a (local) decoder (633) built into the video encoder (603). The decoder (633) reconstructs the symbols to create sample data, in a similar manner as a (remote) decoder would (because any compression between the symbols and the encoded video bitstream is lossless in the video compression technologies of the disclosed invention). The reconstructed sample stream (sample data) is entered into the reference picture memory (634). Since the decoding of the symbol stream produces bit-accurate results independent of the position of the decoder (local or remote), the contents of the reference picture memory (634) will also be the same bit-accuracy for the local encoder and the remote encoder. In other words, the predicted portion of the encoder "sees" as reference picture samples exactly the same sample values as the decoder would "see" if using prediction during decoding. This fundamental principle of reference picture synchronism (and eventually drift if synchronism cannot be maintained, for example due to channel errors) is also used in some related fields of technology.

[0089] A "local" decoder (633) may operate in the same manner as a "remote" decoder, such as the video decoder (510) detailed above with reference to FIG. 5. However, again, referring to FIG. 5, since the symbols are available and encoding/decoding of the symbols into the encoded video sequence by the entropy encoder (645) and analyzer (520) can be lossless, the entropy decoding parts of the video decoder (510) including the buffer memory (515) and the analyzer (520) , may not be fully implemented in the local decoder (633).

[0090] It can thus be concluded that any decoding technology present in a decoder, with the exception of a/entropy decoding analysis, must also necessarily be present, in substantially identical functional form, in the corresponding encoder. For this reason, the disclosed invention is focused on the operation of the decoder. The description of encoding technologies can be abbreviated as they are the inverse of the decoded technologies described in detail. Only in some areas a more detailed description is required, which is given below.

[0091] In some examples, in operation, source encoder (630) may perform motion-compensated predictive coding, in which an input picture is predictively encoded based on one or more previously encoded pictures from a video sequence, referred to as "reference pictures". Thus, the encoding engine (632) encodes the differences between the pixel blocks of the input image and the pixel blocks of the reference image(s) that can be selected as the predicted reference(s) to the input image.

[0092] The local video decoder (633) may decode the encoded video data of pictures that can be specified as reference pictures based on the symbols generated by the source encoder (630). The operations of the encoding engine (632) may be predominantly lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence can typically be a copy of the original video sequence with some errors. The local video decoder (633) duplicates the decoding processes that may be performed by the video decoder on the reference pictures and may cause the reconstructed reference pictures to be stored in the reference picture cache (634). Thus, the video encoder (603) can locally store copies of the reconstructed reference pictures having the same content as the reconstructed reference pictures to be received by the video decoder at the receiver side (in the absence of transmission errors).

[0093] The predictor (635) may perform prediction searches for the encoding engine (632). Thus, for a new picture to be encoded, the predictor (635) can search the reference picture memory (634) for sample data (as candidates for the role of reference pixel blocks) or some metadata, for example, motion vectors of the reference picture, block shapes etc., which can serve as a proper reference for predicting new images. The predictor (635) may operate on a sample block × pixel block basis to find appropriate references for prediction. In some cases, according to the search results obtained by the predictor (635), the input picture may have prediction references extracted from the plurality of reference pictures stored in the reference picture memory (634).

[0094] The controller (650) may control the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters used to encode video data.

[0095] The output signal of all the above functional modules can be entropy encoded in an entropy encoder (645). An entropy encoder (645) translates symbols generated by various functional units into a coded video sequence by lossless symbol compression according to techniques such as Huffman coding, variable run length coding, arithmetic coding, and so on.

[0096] The transmitter (640) may buffer the encoded video sequence(s) created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which may be a hardware/software link to a storage device where encoded video data. The transmitter (640) may combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and/or auxiliary data streams (sources not shown).

[0097] The controller (650) may control the operation of the video encoder (603). During encoding, the controller (650) may assign to each encoded picture an encoded picture type, which may determine the encoding methods applicable to the corresponding picture. For example, images can often be assigned the following image types:

[0098] An intra-picture (I-picture) that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra-pictures, including, for example, pictures in the Independent Decoder Refresh ("IDR" format). The person skilled in the art will be aware of the varieties of I-images and their respective applications and features.

[0099] A predicted picture (P-picture) that can be encoded and decoded using intra prediction or inter prediction using at most one motion vector and a reference index to predict sample values of each block.

[0100] A bi-directionally predicted picture (B picture) that can be encoded and decoded using intra prediction or inter prediction using no more than two motion vectors and reference indices to predict sample values of each block. Likewise, multi-predicted pictures may use more than two reference pictures and associated metadata to reconstruct a single block.

[0101] Source images typically allow for spatial division into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coding on a block-by-block basis. Blocks may be predictively encoded with reference to other (previously encoded) blocks determined by the coding assignment applied to the respective images of those blocks. For example, I-picture blocks may be encoded without prediction or predictively with reference to previously encoded blocks of the same picture (spatial prediction or intra prediction). The pixel blocks of P-pictures may be predictively encoded by spatial prediction or temporal prediction with reference to one previously encoded reference picture. Blocks of B-pictures may be predictively encoded by spatial prediction or temporal prediction with reference to one or two previously encoded reference pictures.

[0102] The video encoder (603) may perform encoding operations according to a predetermined video coding technology or standard, such as ITU-T Rec. H.265. In operation, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. Therefore, the encoded video data may conform to the syntax given by the video coding technology or standard being used.

[0103] According to an embodiment, the transmitter (640) may transmit additional data with encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal/spatial/SNR enhancement layers, other forms of redundant data such as redundant images and slices, SEI messages, VUI parameter set fragments, and so on.

[0104] The video may be captured as a plurality of source images (video images) in time sequence. Intra-image prediction (often abbreviated as intra-image prediction) uses spatial correlation within a given image, and inter-image prediction uses (temporal or otherwise) correlation between images. For example, a specific image to be encoded/decoded, which is called the current image, is divided into blocks. When a block in the current picture is similar to a reference block in a previously encoded and still buffered reference picture in a video, the block in the current picture may be encoded with a vector, which is referred to as a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension identifying the reference picture in case multiple reference pictures are used.

[0105] In some embodiments, a dual prediction technique may be used in inter-picture prediction. According to the double prediction method, two reference pictures are used, for example, a first reference picture and a second reference picture, both of which precede the current picture in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current picture may be encoded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. The block may be predicted by a combination of the first reference block and the second reference block.

[0106] Additionally, a combining mode technique may be used in inter-picture prediction to improve coding efficiency.

[0107] According to some embodiments of the invention, predictions, such as inter-picture predictions and intra-picture predictions, are performed block by block. For example, according to the HEVC standard, an image in a video sequence is divided into coding tree units (CTUs) for compression, CTUs in an image are the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, the CTU includes three coding tree blocks (CTBs), namely, one luma CTB and two chrominance CTBs. Each CTU may recursively divide the quadtree into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU or 4 32x32 pixel CUs or 16 16x16 pixel CUs. For example, each CU is analyzed to determine a prediction type for the CU, such as an inter prediction type or an intra prediction type. The CU is divided into one or more prediction units (PU) depending on temporal and/or spatial predictability. In general, each PU includes a Luma Prediction Unit (PB) and two Chroma PBs. According to an embodiment, an encoding (encoding/decoding) prediction operation is performed in a prediction block unit. Using the luminance prediction block as an example of the prediction block, the prediction block includes a matrix of values (e.g., luminance values) for pixels, e.g., 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc. P.

[0108] In FIG. 7 shows a diagram of a video encoder (703) according to another embodiment of the invention. The video encoder (703) is configured to receive a processing unit (eg, a prediction unit) of sample values in the current video image in the video sequence and encoding the processing unit into an encoded image that is part of the encoded video sequence. For example, video encoder (703) is used instead of video encoder (403) in the example shown in FIG. four.

[0109] In the HEVC example, video encoder (703) receives a matrix of sample values for a processing block, such as an 8x8 sample prediction block, or the like. The video encoder (703) determines whether the processing block is best encoded in intra mode, inter mode, or dual prediction mode using, for example, rate-distortion optimization. When the processing block is to be encoded in intra mode, the video encoder (703) may use an intra prediction technique to encode the processing block into an encoded picture; and when the processing block is to be encoded in inter mode or double prediction mode, the video encoder (703) may use an inter prediction or double prediction technique, respectively, to encode the processing block into an encoded picture. In some video coding technologies, the combining mode may be a sub-mode of inter-picture prediction where a motion vector is derived from one or more motion vector predictors without involving the encoded motion vector component outside of the predictors. In some other video coding technologies, there may be a motion vector component applicable to a given block. By way of example, the video encoder (703) includes other components, such as a mode determination module (not shown) for determining the mode of the processing units.

[0110] In the example shown in FIG. 7, the video encoder (703) includes an inter encoder (730), an intra encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a common controller (721), and an entropy encoder (725). ) connected to each other as shown in Fig. 7.

[0111] The inter-encoder (730) is configured to receive samples of the current block (e.g., processing block), compare the block with one or more reference blocks in reference pictures (e.g., blocks in previous pictures and later pictures), generate external prediction (eg, describing redundant information according to the intra coding method, motion vectors, combining mode information), and calculating inter prediction results (eg, prediction block) based on the inter prediction information using any suitable method. In some examples, reference pictures are decoded reference pictures that are decoded based on encoded video information.

[0112] The intra-encoder (722) is configured to receive samples of the current block (e.g., processing block), in some cases comparing the block with blocks previously encoded in the same picture, generating quantized coefficients after the transform, and in some cases also information intra prediction (eg, intra prediction direction information according to one or more intra coding methods). By way of example, the intra encoder 722 also calculates intra prediction results (eg, prediction block) based on intra prediction information and reference blocks in the same picture.

[0113] The common controller (721) is configured to determine common control data and control other components of the video encoder (703) based on the common control data. For example, the common controller (721) determines the mode of the block and outputs a control signal to the switch (726) based on the mode. For example, when the mode is the intra mode, the common controller (721) controls the switch (726) to select the result of the intra mode for use by the residual calculator (723), and controls the entropy encoder (725) to select the intra prediction information and turn on the intra prediction information into the bitstream; and when the mode is inter mode, the common controller (721) controls the switch (726) to select the inter prediction result to be used by the residual calculator (723), and controls the entropy encoder (725) to select the inter prediction information and include the inter prediction information in the bit flow.

[0114] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction results selected from the intra-coder (722) or inter-coder (730). The residual encoder (724) is configured to operate on the residual data to encode the residual data to generate transform coefficients. For example, the residual encoder (724) is configured to transform residual data from the spatial domain to the frequency domain and generate transform coefficients. Then, the transform coefficients are subjected to quantization processing to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform inverse transformation and generate decoded residual data. The decoded residual data may be appropriately used by the intra-coder (722) and the inter-coder (730). For example, the intercoder (730) may generate decoded blocks based on the decoded residual data and inter prediction information, and the intra encoder (722) may generate decoded blocks based on the decoded residual data and intra prediction information. The decoded blocks are properly processed to generate decoded pictures, and the decoded pictures can be buffered in a memory circuit (not shown) and used as reference pictures in some examples.

[0115] The entropy encoder (725) is configured to format the bitstream so that it includes a coded block. The entropy encoder (725) is configured to include various information according to an appropriate standard, such as the HEVC standard. For example, the entropy encoder (725) is configured to include common control data, selected prediction information (eg, intra prediction information or inter prediction information), residual information, and other appropriate information in the bitstream. Note that, according to the disclosed invention, when encoding a block in the combining sub-mode of any of the inter-mode and the dual-prediction mode, there is no residual information.

[0116] In FIG. 8 shows a diagram of a video decoder (810) according to another embodiment of the invention. The video decoder (810) is configured to receive encoded pictures constituting part of the encoded

video sequence, and decoding the encoded pictures to generate reconstructed pictures. For example, video decoder (810) is used instead of video decoder (410) in the example shown in FIG. four.

[0117] In the example shown in FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction unit (874), and an intra decoder (872) connected to each other as shown in FIG. eight.

[0118] The entropy decoder (871) may be configured to reconstruct, from the encoded image, some symbols that represent the syntax elements that make up the encoded image. Such symbols may include, for example, a block coding mode (e.g., intra mode, inter mode, dual prediction mode, the latter two being in a pooling submode or other submode), prediction information (e.g., intra prediction information, or inter prediction information ) that can identify a specific sample or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients, and the like. For example, when the prediction mode is an intra or dual prediction mode, the inter prediction information is input to the inter decoder (880); and when the prediction type is an intra prediction type, the intra prediction information is input to the intra decoder (872). The residual information may be inverse quantized and fed to the residual decoder (873).

[0119] The inter decoder (880) is configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0120] The intra decoder (872) is configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0121] The residual decoder (873) is configured to perform inverse quantization to extract the dequantized transform coefficients and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require some control information (include a quantizer parameter (QP, Quantizer Parameter)), and this information may be provided by the entropy decoder (871) (the data path is not shown since this may only be small volume control information).

[0122] The reconstruction module (874) is configured to combine, in the spatial domain, the residual at the output of the residual decoder (873) and prediction results (at the output of the inter- or intra-prediction modules, as appropriate) to form a reconstructed block that may be included into the reconstructed image, which, in turn, can be part of the reconstructed video. Note that other suitable operations, such as a release operation and the like, may be performed to improve the visual quality.

[0123] Note that video encoders (403), (603) and (703) and video decoders (410), (510) and (810) may be implemented using any suitable technique. According to an embodiment, video encoders (403), (603) and (703) and video decoders (410), (510) and (810) may be implemented using one or more integrated circuits. In another embodiment, video encoders (403), (603) and (603) and video decoders (410), (510) and (810) may be implemented using one or more processors that execute program instructions.

[0124] II. Transform processing methods

[0125] 1. A block splitting structure including a quadtree splitting structure

[0126] The block splitting structure may be referred to as a coding tree. In some embodiments, through the use of a quadtree structure, a coding tree unit (CTU) is divided into coding units (CU) to adapt to different local characteristics. The decision whether to encode an image area using inter-picture (temporal) or intra-picture (spatial) prediction is made at the CU layer. Each CU may be further divided into one, two, or four prediction units (PU) according to the division type of the PU. Within one PU, the same prediction process is applied and meaningful information is transmitted to a decoder based on the PU.

[0127] Having obtained a residual block using a prediction process based on the division type of the PU, the CU can be partitioned into transform units (TUs) according to another quadtree structure. As can be seen, there are different partitioning principles, including CU, PU, and TU. In some embodiments, the CU or TU may only be square, while the PU may be square or rectangular. In some embodiments, one coding block may be further divided into four square subblocks, and the transformation may be performed on each subblock, i. e. T.U. Each TU can be further recursively divided into smaller TUs by using a quadtree structure called a residual quadtree (RQT).

[0128] At the image boundary, in some embodiments, implicit quadtree partitioning can be used, which allows the quadtree block to continue splitting until its size fits within the image boundary.

[0129] 2. Quadtree plus binary tree (QTBT) block partitioning structure

[0130] In some embodiments, a quadtree plus binary tree (QTBT) structure is used. The QTBT framework eliminates the concepts of multiple partition types (the concepts of CU, PU, and TU), and supports additional flexibility for CU partition forms. In the block structure of the QTBT, the CU can be either square or rectangular.

[0131] In FIG. 9A shows a CTU (910) that is partitioned using the QTBT structure (920) shown in FIG. 9B. First, the CTU (910) is partitioned according to the quadtree structure. The edge nodes of the quadtree are further split according to the binary tree structure or the quadtree structure. In binary tree partitioning, there can be two types of partitioning, symmetrical horizontal partitioning and symmetrical vertical partitioning. The leaf nodes of a binary tree are called CUs, which can be used for prediction and transformation processing without any additional partitioning. Accordingly, CUs, PUs, and TUs in a QTBT coding block structure have the same block size.

[0132] In some embodiments, a CU may include coding units (CBs) of different color components. For example, one CU contains one luma CB and two chrominance CBs in the case of 4:2:0 chroma P and B slices. The CU may include the CB of a single color component. For example, one CU contains a single luminance CB, or only two chrominance CBs in the case of I-slices.

[0133] In some embodiments, the following parameters are set for the QTBT partitioning scheme:

- CTU size: the size of the root node of the quadtree, for example, the same principle as in HEVC.

- MinQTSize: The minimum allowed size of a quadtree edge node.

- MaxBTSize: maximum allowed size of the root node of the binary tree.

- MaxBTDepth: maximum allowed binary tree depth.

- MinBTSize: The minimum allowed size of the edge node of the binary tree.

[0134] In one example of a QTBT partitioning structure, the CTU size is set to 128x128 luma samples with two corresponding blocks of 64x64 chrominance samples, MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize (for both width and height ) is set to 4×4, and MaxBTDepth is set to 4. First, quadtree partitioning is applied to the CTU to generate edge quadtree nodes. The edge nodes of the quadtree may have a size from 16x16 (ie MinQTSize) to 128x128 (ie CTU size). If a quadtree edge node has a size of 128x128, it will not be further divided as a binary tree because the size exceeds MaxBTSize (ie 64x64). Otherwise, the edge node of the quadtree may be further split as a binary tree. Therefore, the edge node of the quadtree is also the root node for the binary tree, and it has a binary tree depth of 0.

[0135] When the depth of the binary tree reaches MaxBTDepth (ie 4), no further division is provided. When a binary tree node has a width equal to MinBTSize (ie 4), no additional horizontal split is provided. Similarly, when a binary tree node has a height equal to MinBTSize, no additional vertical split is provided. The edge nodes of the binary tree are further processed through prediction and transformation processing without any further splitting. According to an embodiment, the maximum CTU size is 256×256 luminance samples.

[0136] FIG. 9A and 9B, solid lines indicate quadtree partitioning and dotted lines indicate binary tree partitioning. At each split (ie, non-edge) of a binary tree node, one flag is signaled to indicate the type of split to use (ie, horizontal or vertical). For example, 0 indicates a horizontal split and 1 indicates a vertical split. For a quadtree split, it is not necessary to specify a split type, because a quadtree split always divides a block both horizontally and vertically to create 4 equally sized subblocks.

[0137] In some embodiments, the QTBT schema maintains the flexibility that luma and chrominance have separate QTBT structures. For example, for P and B slices, the luma and chrominance units in the same CTU share the same QTBT structure. However, for I-slices, the luma CTB is partitioned into CUs according to the QTBT structure, and the chrominance blocks are partitioned into chroma CUs by another QTBT structure. Thus, a CU in an I slice consists of a luma component coding block or two chroma component coding blocks, and a CU in a P or B slice consists of all three color component coding blocks.

[0138] In some embodiments, inter-prediction for small blocks is limited to reduce motion compensation memory access. For example, dual prediction is not supported for 4x8 and 8x4 blocks, and inter prediction is not supported for 4x4 blocks.

[0139] 3. Trinity Tree (TT) Block Break Structure

[0140] In some embodiments, a multi-type tree (MTT) structure is used to partition an image. The MTT structure is a more flexible tree structure than the QTBT structure. In addition to the quadtree and binary tree, the MTT uses the horizontal center-side ternary tree and the vertical center-side ternary tree shown in FIG. 9C and FIG. 9D, respectively. Ternary tree partitioning can complement quadtree and binary tree partitioning. For example, a ternary split tree can capture objects that are in the center of a block, while a quadtree and a binary tree can split by crossing the centers of the blocks. The width and height of the partitions obtained by ternary trees are powers of 2, which eliminates the need for additional transformation partitioning.

[0141] In the example, the two-level tree design is mainly motivated by the reduction in complexity. For example, the tree traversal complexity is T ^D , where T is the number of split types and D is the depth of the tree.

[0142] 4. Primary Conversion Examples

[0143] In some embodiments, such as HEVC, 4-point, 8-point, 16-point, and 32-point DCT-2 transforms are used as primary transforms. In FIG. 10A-10D show the transform kernel matrices of the 4-point, 8-point, 16-point, and 32-point DCT-2, respectively. The elements of these transformation kernel matrices can be represented using 8-bit integers, and therefore these transformation kernel matrices are referred to as 8-bit transformation kernels. As shown, the transformation kernel matrix of the smaller DCT-2 is part of the matrix of the larger DCT-2.

[0144] DCT-2 core matrices exhibit symmetry/antisymmetry characteristics. Accordingly, a so-called "partial butterfly" implementation can be supported to reduce the number of operation counters (multiplications, additions/subtractions, shifts). The identical results of matrix multiplication can be obtained using the partial butterfly implementation.

[0145] 5. Additional Primary Transform Examples

[0146] In some embodiments, in addition to the 4-point, 8-point, 16-point, and 32-point DCT-2 transformations described above, additional 2-point and 64-point DCT-2 transformations are used. In FIG. 11A-11E show a 64x64 matrix of a 64-point DCT-2 transform kernel.

[0147] In some embodiments, in addition to the DCT-2 and 4x4 DST-7 transforms, an adaptive multiple transform (AMT) (also known as enhanced multiple transform (EMT) is used to encode the remainder of both inter- and intra-coded blocks). multiple transform), or the choice of multiple transformations (MTS, multiple transform selection)). The AMT uses multiple selected transforms from the Discrete Cosine Transform (DCT) / Discrete Sine Transform (DST) families in addition to the DCT-2 transform, such as the DST-7 transform kernel matrices or the DCT-8 transform. In FIG. 12 shows the basis transform functions from the selected DST/DCT transforms.

[0148] In some embodiments, the DST/DCT transform kernel matrices used in the AMT are in 8-bit format. In some embodiments, an AMT is applied to a CU where u, width, and height are less than or equal to 32. Whether or not an AMT is applied may be controlled by a flag (eg, mts_flag). For example, when mts_flag is 0, only DCT-2 is used to encode the residual block. When mts_flag is 1, an index (eg, mts_idx) may be further signaled using 2 bins to indicate the horizontal and vertical transformations to be used.

[0149] FIG. 13 is a table (1300) illustrating the mapping relationship between an index value (eg, mts_idx) and corresponding horizontal or vertical transformations. Line (1301) where mts_idx is -1 corresponds to a scenario where the flag (eg mts_flag) is 0 and DCT-2 is used. Rows (1302)-(1305) where mts_idx is 0, 1, 2, or 3 correspond to the scenario where mts_flag is 1. In the two right columns of table (1300), 0 denotes the DCT-2 transform type, 1 denotes the DST- 7 and 2 denotes the type of DCT 8 conversion.

[0150] In FIG. 14A-14D show DST-7 transform kernel matrices. In FIG. 15A-15D show DCT-8 transform kernel matrices.

[0151] 6. Intra-subdivision (ISP) coding mode

[0152] In some embodiments, an intra-subdivision (ISP) coding mode is used. In the ISP coding mode, the intra-predicted luma block can be split vertically or horizontally into 2 or 4 subparts. The number of subparts may depend on the size of the block. In FIG. 16 shows the number of sub-parts depending on the block size. In FIG. 17 shows an example where a 4x8 or 8x4 block is split into two sub-parts. In FIG. 18 shows an example where a block larger than 4×8 or 8×4 is divided into four sub-parts. In the example, all subparts satisfy the condition of having at least 16 samples. In the example, ISP does not apply to chrominance components.

[0153] In the example, for each of the subparts into which the coding block is divided, a residual signal is generated by entropy decoding the respective coefficients sent from the encoder and then dequantizing and inverse transforming them. Then, the first of the subparts is internally predicted to generate a prediction signal. The prediction signal is added to the corresponding first split residual signal to obtain the corresponding reconstructed samples. The reconstructed sample values of the first split may then be available to generate a prediction of the second of the subparts. This process can be repeated split by split until all subparts from the coding block have been reconstructed. In the example, all subparts share the same intra mode.

[0154] In an embodiment, the ISP encoding mode is only tested with intra modes that are part of the most probable mode (MPM) list. Accordingly, if the block uses an ISP, then the MPM flag may be assumed to be one. In addition, when an ISP is used for a particular block, the corresponding MPM list will be modified to exclude the DC mode and to give priority to horizontal intra modes for horizontal ISP separation and vertical intra modes for vertical.

[0155] In the ISP coding mode, each partition can be considered as a TU since the transformation and reconstruction are performed separately for each sub-part.

[0156] FIG. 19A-19B show an example of syntax elements (1900) signaled for ISP coding mode. As shown in frame (1910), a syntax element such as intra_subpartitions_mode_flag indicates whether an ISP is being used. A syntax element, such as intra_subpartitions_split_flag, specifies the splitting direction (vertical or horizontal).

[0157] 7. Subblock Transform (SBT)

[0158] In some embodiments, a subblock transform (SBT), also referred to as a spatially varying transform (SVT), is used. SBT can be applied to inter prediction residuals. In some examples, the residual block is included in the coding block and is smaller than the coding block. Thus, the size of the transform in SBT is smaller than the size of the coding block. For the area not covered by the residual block, zero residual can be assumed, and thus no conversion processing is performed.

[0159] In FIG. 20A-20D show sub-block types (SVT-H, SVT-V) (e.g., horizontal or vertical splits), sizes, and positions (e.g., left half, left quarter, right half, right quarter, top half, top quarter, bottom half , lower quarter) supported in SBT. The shaded areas labeled "A" are residual blocks with transformation, and other regions are assumed to be zero residual without transformation.

[0160] By way of example, in FIG. 21A-21I show changes made to the specification text of a video coding standard (eg, VVC) when using SBT. Added texts are shown in frames (2101)-(2113). As shown, additional syntax elements, such as additional overhead bits cu_sbt_flag, cu_sbt_quad_flag, cusbthorizontalHag, and cu_sbt_pos_flag, may be signaled to indicate the sub-block type (horizontal or vertical), size (half or quarter), and position (left, right, top, or bottom), respectively.

[0161] 8. YUV formats

[0162] In FIG. 22 shows different YUV formats (eg, 4:4:4, 4:2:2, 4:1:1, and 4:2:0) used in some embodiments. In the example, intra-prediction based on a cross-component linear model is used for a 4:2:0 format. To obtain a decimated luminance sample corresponding to the chrominance sample, as shown in FIG. 22, a six-tap interpolation filter can be applied. In the example, the decimated luminance sample Rec' _L [x, y] can be computed from nearby reconstructed luminance samples (denoted Rec _L [x, y]), as follows:

Rec' _L [x, y]=(2×Rec _L -[2x, 2y]+2×Rec _L [2x, 2y+1]+

Rec _L [2x-1,2y]+Rec _L [2x+1,2y]+

Rec _L [2x-1,2y+l]+Rec _L [2x+1,2y+1]+4)>>3

The decimated luma sample Rec' _L [x, y] can be used to predict the chrominance sample using the cross-component linear model mode.

[0163] 9. Virtual Pipeline Data Unit (VPDU)

[0164] Virtual Pipeline Data Units (VPDUs) can be defined as non-overlapping units of luma (L) MxM / chrominance (C) NxN in an image. In some decoder hardware implementations, successive VPDUs are processed simultaneously by multiple pipeline stages. Different stages simultaneously process different VPDUs. The size of the VPDU may be approximately proportional to the size of the buffer in the stages of the pipeline, so it is desirable to keep the size of the VPDU at a certain level (eg, 64x64 or less). In some decoders, the VPDU size is set to the maximum Transform Unit (TU) size. Increasing the maximum TU size from 32x32-L/16xl6-C in HEVC to 64x64-L/32x32-C in modern VVC can result in a coding gain, which is expected to result in a 4x increase in VPDU size compared to HEVC. However, the BT and TT structures that VVC uses to achieve additional coding gains can be recursively applied to 128x128-L/64x64-C coding tree blocks, resulting in a 16-fold increase in VPDU size (128x128- L/64×64-C) compared to HEVC.

[0165] FIG. 23 shows some forbidden partitions of TT and BT.

[0166] In order to maintain a VPDU size of 64x64 luma samples, in some embodiments, some partitioning constraints apply (with syntax signaling modification):

- TT split is not allowed for CU when width or height or both width and height are 128.

- for CU 128×N with N≤64 (i.e. when the width is 128 and the height is less than 128), the horizontal division of the BT is not allowed.

- for N×128 CUs with N≤64 (i.e. when the height is 128 and the width is less than 128), vertical separation of the BT is not allowed.

[0167] III. Transform block splitting and processing methods

[0168] In some embodiments, a fixed maximum allowable transform unit (TU) size or maximum TU size (eg, 64x64 pixels or samples) is used. In some embodiments, adjustable or configurable maximum TU sizes are used, since the maximum TU size may affect the complexity of the hardware, for example, for encoder implementation (eg, pipeline intermediate buffer size, number of multipliers, etc.). For example, in addition to the size of 64x64 samples, the maximum TU size can take on other values, such as 32x32 samples, 16x16 samples, and the like.

[0169] In some video standards, SBT and ISP may be used. For example, in SBT, an SPS flag, such as sps_sbt_max_size_64_flag, is signaled to indicate whether the largest SBT size is length 32 or length 64. When sps_sbt_max_size_64_flag is TRUE (i.e., the largest SBT size is length 64), and the maximum TU size is 32 point, encoder failure may be triggered. In general, the size "length L" or "L dots" means the maximum size of CU, TU, CB, TB, VPDU, etc. For example, when the maximum size of a TU is 32 dots or the length is 32, the width and height of the TU are less than or equal to 32.

[0170] In some embodiments, the ISP mode is enabled for various CU sizes, however, when the maximum TU size is set to less than 64, there may be a conflict as to whether splitting with implicit translation or splitting explicitly using ISP with signaling. For example, when the maximum TU size is 16, for a 64x16 CU without an ISP, the CU may be implicitly divided into four 16x16 TUs. With an ISP, a 64x16 CU can share with a vertical ISP and thus can be divided into four 16x16 TUs, but using signaling.

[0171] When the maximum TU size is less than 64x64, the processing order of the TU needs to be coordinated with the VPDU implementation.

[0172] The embodiments described herein may be used individually or combined in any order. Additionally, embodiments may be implemented by processing circuitry (eg, one or more processors or one or more integrated circuits) in an encoder, decoder, or the like. In one example, one or more processors execute a program that is stored on a computer-readable medium.

[0173] In the description of the invention, the high-level syntax element (HLS, high-level syntax) can mean a video parameter set (VPS, Video Parameter Set), a sequence parameter set (SPS, Sequence Parameter Set), a picture parameter set (PPS, Picture Parameter Set), slice header, tile header, tile group header, etc. The CTU header may refer to the syntax element(s) signaled to the CTU, for example, as header information. In the example, the CTU size is the maximum CU size.

[0174] In general, when the luminance size (represented by luminance samples) of a particular unit (eg, TU, CU) is known, a corresponding chrominance size can be obtained, which is given by the number of chrominance samples. In the example, the YUV format is 4:2:0 and the CU has a CU size of 64x64 luma samples (or 64x64-L). Accordingly, the CU has a CU size of 32×32 chroma samples (or 32×32-C). The CU size may be referred to as 64x64-L, 32x32-C, or 64x64-L/32x32-C. Similarly, a TU has a TU size of 64x64 luminance samples (or 64x64-L). Accordingly, the TU has a TU size of 32x32 chroma samples (or 32x32-C). The TU size may be referred to as 64x64-L, 32x32-C, or 64x64-L/32x32-C. For example, a TU includes a luma conversion (TB) unit and two chrominance TVs. The luminance TV has a size of 64x64-L, and each of the chrominance TVs has a size of 32x32-C. In general, the embodiments and methods described for CU or TU can be appropriately adapted to CB and TV, respectively.

[0175] The CU may include a 64x64-L luma block and two 32x32-C chrominance blocks. In the following description, the size of a TU is represented using the luminance samples in the TU. For example, the maximum size TU M samples means the maximum size TU M luma samples. Likewise, other sizes, such as VPDU size and CU size, are also represented using the respective luminance samples in appropriate units, such as VPDU and CU, respectively. Of course, TU size, VPDU size, CU size, etc. can be represented using chrominance samples or a combination of luma and chrominance samples.

[0176] The block size may refer to the width, height, or area of the block. For example, the maximum TU size may refer to the width, height, or area of the maximum TU. In general, TU, CU, VPDU, etc. may be any suitable shape, including rectangular, square, L-shaped, or any suitable shape. In the case of an irregularly shaped block, such as an L-shape, the block size may indicate the area of the block.

[0177] In some embodiments, the VPDU size and/or the maximum TU size may be signaled in the coded video bitstream, such as in SPS and PPS. As described above, the VPDU size and/or maximum TU size may be signaled in luma samples. Alternatively, the VPDU size and/or the maximum TU size may be signaled in the chrominance samples.

[0178] In some embodiments, the VPDU size and/or maximum TU size may be stored at the encoder and/or decoder, so the VPDU size and/or maximum TU size is not signaled. In one example, the VPDU size and/or the maximum TU size may be stored in the profile and/or layer definitions. The VPDU size and/or maximum TU size may be stored as luminance or chrominance samples.

[0179] In some embodiments, VPDUs share the same size but may have different shapes. For example, when the size of the VPDU is 4096 luminance samples, the VPDU may be in the shape of a 64x64 square or a 32x128 rectangle. The VPDU may also have other shapes, such as L-shape, as long as the size of the VPDU is 4096 luminance samples. The above description also applies to certain TUs.

[0180] 1. Example A

[0181] According to aspects of the invention, the maximum allowable TU size (also referred to as the maximum TU size) is M samples (eg, MxM sample size). In the example, the maximum width and maximum height of the TU is M. In the example, the maximum area of the TU is M×M. The processing data unit size (eg, VPDU size) is K samples (eg, K×K samples). In the example, the maximum width and maximum height of the processing data unit is K. In the example, the maximum area of the processing data unit is K×K. A W×H CU has a width of W samples and a height of H samples. A CU may be partitioned into multiple sub-units, referred to as sub-processing units (SPUs), based on the size of the CU and the size K of the processing data unit. The CU may be partitioned into SPUs using any suitable partitioning structure, or a combination of any suitable partitioning structures, such as QTBT, QT, BT, TT, or combinations thereof. SPUs can be the same size or different sizes.

[0182] According to an embodiment, a CU is partitioned into SPUs when the width W or the height H is greater than K. In the example, the SPUs are the same size (i.e., SPU size), and each SPU has a size of Min(W, K) × Min (H, K) samples. Thus, the width of each SPU is the minimum of W and K, and the height of the SPU is the minimum of H and K. In some examples, prior to splitting the CU, it may be determined whether to split the CU based on the size of the CU and the size of K of the processing data unit.

[0183] SPUs in a CU can be further divided into TUs of size, for example, M×M samples. In some examples, the SPU may be partitioned into TUs of size Min(W, K, M) × Min(H, K, M). In some examples, prior to splitting the SPU, it may be determined whether to split the SPU based on the size of the SPU and the maximum TU size M. The SPU may be split using any suitable split structure or combination of any suitable split structures, e.g., QTBT, QT, BT, TT or their combinations. According to aspects of the invention, one or more partition structures for partitioning an SPU may be determined based on the size of the SPU and the maximum TU size M. In an example, the SPU may be recursively divided into TUs using the defined one or more partition structures.

[0184] In an example, when the width and height of the SPU is larger than the maximum TU size M, the SPU is divided into TUs of size MxM using a quadtree partitioning structure. SPUs can be recursively divided into TUs using a quadtree partitioning structure.

[0185] In an example where the width of the SPU is greater than M and the height of the SPU is M, the SPU is divided into M×M TUs using a binary tree vertical partition structure. For example, M is 32 and the SPU has a size of 64x32. Thus, the width of the SPU is 64 and the height of the SPU is 32. Accordingly, a binary tree vertical partition structure can be used to partition the SPU into two 32x32 TUs. SPUs can be recursively divided into TUs using a vertical partitioning structure in the form of a binary tree.

[0186] In an example where the SPU width is greater than M and the SPU height is less than M, the SPU may be divided into TUs using a binary tree vertical partitioning structure where the TU width is M and the TU height is equal to the SPU height.

[0187] In an example where the height of the SPU is greater than M and the width of the SPU is M, the SPU is divided into M×M TUs using a horizontal binary tree partition structure. SPUs can be recursively divided into TUs using a horizontal binary tree partitioning structure. For example, M is 32 and the SPU is 32×64. Thus, the width of the SPU is 32 and the height of the SPU is 64. Accordingly, a horizontal binary tree partition structure can be used to partition the SPU into two 32x32 TUs.

[0188] When the height of the SPU is greater than M and the width of the SPU is less than M, the SPU may be divided into TUs using a horizontal binary tree partitioning structure, where the height of the TU is equal to M and the width of the TU is equal to the width of the SPU.

[0189] The syntax of the transformation tree in FIG. 24 shows an example of splitting the SPU, and the processing order used to process the TU.

[0190] In the example, a W×H CU can be split in two steps. In the first step, the CU is partitioned into SPUs, where each SPU has the size Min(W, K) × Min(H, K). Then, in a second step, each SPU is further partitioned into TUs, where each TU has a size of MxM.

[0191] When processing TUs in a CU, SPUs in a CU may be scanned and processed in a first scan order (also referred to as first order). Additionally, in each of the SPUs, TUs may be scanned and processed in second order (also referred to as second order).

[0192] In various embodiments, the first order for processing the SPU may be a raster scan order, a vertical scan order (eg, scans the SPU in columns from left to right or vice versa), a zigzag order, a diagonal scan order, and the like.

[0193] In various embodiments, the second order for processing the TUs in each SPU may be raster scan order, vertical scan order (e.g., scans the TU in columns from left to right or vice versa), zigzag order, diagonal scan order, and the like.

[0194] The first order and the second order may be the same or different in different embodiments. For example, the first order for SPU processing and the second order for TV processing in each of the SPUs are the raster scan order according to the embodiment.

[0195] 2. Example B

[0196] FIG. 25 shows a CU (2510) with sample size WxH where W=128 and H=64. The maximum TU M size is 32 samples. The processing data unit size K, eg the VPDU size, is 64 samples. First, CU (2510) is divided into a first SPU 64×64 (2520) and a second SPU 64×64 (2530). Then, the first SPU (2520) and the second SPU (2530) may be further partitioned by HaTU 0-7, each having a size of MxM samples. TUs 0-3 are included in the first SPU (2520) and TUs 4-7 are included in the second SPU (2530).

[0197] According to the first order, first the first SPU (2520) can be processed, and then the second SPU (2530). In the first SPU (2520) or the second SPU (2530), the second order used to process TUs 0-3 or 4-7 is the raster scan order. Accordingly, TUs 0-7 are processed in the order indicated by the arrow (2551). The first order and/or the second order may be specified explicitly (eg, via encoder-to-decoder signaling) or implicitly.

[0198] In some examples, partitioning a CU into SPUs, where each of the SPUs further includes a TU, as described above, improves coding efficiency. According to FIG. 25, in the example, the first SPU (2520) is the first VPDU and the second SPU (2530) is the second VPDU. Each of the first VPDU (or first SPU (2520)) and the second VPDU (or second SPU (2530)) may sequentially pass through a multi-stage pipeline, including a first stage (eg, entropy decoding), a second stage (eg, dequantization), the third stage (for example, inverse transformation), etc. According to the first order shown in FIG. 25, the first SPU (2520) should be processed before the second SPU (2530), so the first SPU (2520) is processed in the first stage and then enters the second stage. In an example, when the first SPU (2520) is processed in the second stage, the second SPU (2530) is processed in the first stage to improve coding efficiency. The first SPU (2520) then enters the third stage and the second SPU (2530) may enter the second stage. When the first SPU (2520) is processed in the third stage, the second SPU (2530) may be processed in the second stage. The above description is based on the use of VPDU and multi-stage pipeline by way of example, and may be appropriately adapted to other video coding architectures or methods. The above description can be adapted to the case where the first SPU (2520) is included in the first VPDU and the second SPU (2530) is included in the second VPDU. At least part of the SPU processing in the different stages is performed simultaneously.

[0199] As described above, when the size of an SPU is larger than the size of a TU, a plurality of TUs in a CU may be grouped into a processing unit, such as an SPU (or VPDU), where the SPUs may be processed in a multi-stage pipeline that allows parallel processing (or simultaneous processing) serial SPUs. In some examples, the description can be modified as follows: CU is broken down into first units. Additionally, each of the first units can be broken down into second units. Each of the second units can be divided into third units. In the example, the size of the first units is larger than the size of the second units, and the size of the second units is larger than the size of the third units. Such a partition can be useful when the first multistage pipeline is nested within the second multistage pipeline.

[0200] 3. Example C

[0201] In FIG. 26A shows a CU (2610A) with WxH samples where W=128 and H=32. The maximum size of TU M is 16 samples. The processing data unit size K, eg the VPDU size, is 64 samples. The minimum of W and K is 64, while the minimum of H and K is 32. Thus, the SPU size may be determined to be 64x32 samples, for example, to align transform units with VPDUs. CU (2610A) can be divided into left SPU (2620A) and right SPU (2630A), each of which has a size of 64×32 samples. The two SPUs (2620A) and (2630A) can be scanned and processed in order from left to right.

[0202] Each of the two SPUs (2620A) and (2630A) may be further divided into TUs, each having a maximum TU size of 16×16 samples. As shown, the left SPU (2620A) is split into TUs 0-7 while the right SPU (2630A) is split into TUs 8-15. In the SPU (2620A), TUs 0-7 can be processed in raster scan order. In the SPU (2630A), TUs 8-15 can be processed in raster scan order. Accordingly, TU 0-15 may be scanned and processed in the order indicated by the arrows (2651A), where TU 0 is processed first and TU 15 is processed after TU 0-14 is processed.

[0203] 4. Example D

[0204] In FIG. 26B shows a CU (2610V) of WxH samples where W=128 and H=32. The maximum size of TU M is 16 samples. The processing data unit size K, eg the VPDU size, is 64 samples. Similar to the example shown in FIG. 25, CU (2610V) can be split into two SPUs (2620V) and (2630V), each of which can be further split into TUs. SPUs (2620B) and (2630B) can be processed from left to right in the same order as in FIG. 25. However, unlike the example shown in FIG. 25, TUs 0-7 in SPU (2620V) are processed in zigzag scan order, and TUs 8-15 in SPU (2630V) are processed in zigzag scan order.

[0205] 5. Example E

[0206] In FIG. 27 is a flowchart showing a process (2700) for splitting and processing a transform block according to an embodiment of the invention. Process (2700) may be used in reconstructing a block encoded in intra mode or inter mode. In various embodiments, the process (2700) is performed by a processing circuit, for example, a processing circuit at terminal devices (210), (220), (230), and (240), a processing circuit that performs the functions of a video encoder (403), a processing circuit, which performs the functions of a video decoder (310), a processing circuit that performs the functions of a video decoder (410), a processing circuit that performs the functions of a video encoder (603), etc. In some embodiments, the process (2700) is performed in program instructions, thus, when the processing circuit executes the program instructions, the processing circuit executes the process (2700). The process starts at step (S2701) and proceeds to step (S2710).

[0207] In step (S2710), the encoded CU information in the image may be decoded from the encoded video bitstream. The encoded information may indicate the width W of the samples and the height H of the CU samples.

[0208] In step (S2720), the CU may be partitioned into SPUs, for example, when at least one of the width W and the height H of the CU is larger than the size K of the processing data unit, for example, as described with reference to FIG. 24-26. Based on the size K of the processing data unit and the size of the CU, the size of the SPU can be determined. The width of the SPU may be the smallest of W and K, and the height of the SPU may be the smallest of H and K. Accordingly, the CU may be partitioned into SPUs, each of which has a certain width and height. For example, when W is 128, H is 64, and K is 64, the CU may be divided into a 64x64 first SPU and a 64x64 second SPU. For example, the processing unit may be a VPDU, and thus K may be the size of the VPDU.

[0209] In step (S2730), one or more partition structures for partitioning each of the SPUs may be determined, for example, based on one or a combination of the width and height of the SPU and the maximum TU size of M samples. In the example, at least one of the width and height of the SPU is greater than M.

[0210] As described above, any suitable partitioning structure may be used to partition or partition each of the SPUs. In an example, when the width and height of the SPU is greater than M, one or more partition structures are defined as a quadtree partition structure. In an example, when the width of the SPU is greater than M and the height of the SPU is not greater than M, one or more split structures are defined as a vertical split structure in the form of a binary tree. In an example where the height of the SPU is greater than M and the width of the SPU is not greater than M, one or more partition structures are defined as a horizontal binary tree partition structure.

[0211] In step (S2740), each of the SPUs may be partitioned into TUs based on the determined one or more partition structures. In the example, the corresponding SPU may be recursively partitioned into TUs using one or more partition structures defined.

[0212] In step (S2750), the TUs obtained by splitting the SPUs are processed according to the processing order. For example, the SPUs may be processed according to the first order, and the TUs in each of the SPUs may be processed according to the second order, as described above. The residual data of each TU may be determined by various decoding operations (eg, entropy decoding of transform coefficients, inverse quantization or dequantization, inverse transform, etc.). The process (2700) may proceed to (S2799) and end.

[0213] The process (2700) is described using the CU by way of example. Process (2700) may be appropriately adapted for the CB, for example, a luma block, a chrominance block, and the like. For the sake of brevity, the description in relation to the CB is omitted.

[0214] The process (2700) can be appropriately adapted. For example, one or more of the steps may be modified, omitted, or combined. For example, steps (S2730) and (S2740) may be combined into a single step. Additional step(s) may also be added. The execution order of the process (2700) can also be modified.

[0215] IV. computer system

[0216] The above methods may be implemented in computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, in FIG. 28 shows a computer system (2800) suitable for implementing some embodiments of the disclosed invention.

[0217] Computer software may be coded using any suitable machine code or computer language that may be assembled, compiled, linked, or similarly processed to create code containing instructions that can be executed directly, or through interpretation, execution of microcode, etc. .p., one or more computer central processing units (CPU, central processing unit), graphic processors (GPU, Graphics Processing Unit), etc.

[0218] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and the like.

[0219] The computer system components (2800) shown in FIG. 28 are illustrative in nature and are not intended to impose any limitation on the scope of application or functionality of computer software implementing embodiments of the present invention. The configuration of the components should also not be interpreted as having any dependency or requirement in connection with any one or combination of the components shown in the illustrative embodiment of the computer system (2800).

[0220] The computer system (2800) may include some human interface input devices. Such an input device may be responsible for input by one or more users through, for example, tactile input (e.g., keystrokes, waving, glove movements), audio input (e.g., voice, claps), visual input (e.g., gestures), olfactory input (not shown). Interface devices may also be used to capture certain media not necessarily directly related to conscious human input, such as sound (e.g. speech, music, external sound), images (e.g. scanned images, photographic images received from a still camera), video (eg, 2D video, 3D video including stereoscopic video).

[0221] Human interface input devices may include one or more of (one each depicted): keyboard (2801), mouse (2802), touch pad (2803), touch screen (2810), information glove (not shown). ), joystick (2805), microphone (2806), scanner (2807), camera (2808).

[0222] The computer system (2800) may also include some human interface output devices. Such output devices may stimulate the senses of one or more users through, for example, tactile output, sound, light, and smell/taste. Such output devices may include tactile output devices (e.g., haptic feedback via touch screen (2810), data glove (not shown) or joystick (2805), but may also be tactile feedback devices that do not serve as input devices) , audio output devices (for example: speakers (2809), headphones (not shown)), visual output devices (for example, screens (2810), including CRT screens, LCD screens, plasma screens, OLED screens, each with touch screen input capabilities, each with or without tactile feedback capabilities, some of which are capable of 2D visual output or more than 3D output through, for example, stereographic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)) and printers (not shown).

[0223] The computer system (2800) may also include human accessible storage devices and associated media, such as optical media including CD/DVD ROM/RW (2820) with CD/DVD media (2821), etc. .p., flash memory cards (2822), removable hard drive or solid state drive (2823), conventional magnetic media such as tape and floppy disk (not shown), dedicated ROM/ASIC/PLD based devices such as security hardware keys (not shown) and the like.

[0224] Those skilled in the art should also understand that the term "computer-readable media" as used in connection with the invention disclosed herein does not encompass transmission media, carrier waves, or other short-term signals.

[0225] The computer system (2800) may also include an interface to one or more communication networks. Networks can be, for example, wireless, wired, optical. Networks may additionally be local, wide area, metropolitan, transport and industrial, real-time, latency, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., digital wide area networks for wired or wireless television, including cable television, satellite TV and terrestrial TV, transport and industrial include CANBus, etc. Some networks typically require external network interface adapters that are connected to some general purpose data ports or peripheral buses (2849) (for example, computer system USB ports (2800)); others are typically built into the core of the computer system (2800) by connecting to a system bus as described below (eg, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2800) can communicate with other entities. Such communication may be unidirectional with receive-only capability (eg broadcast), unidirectional with transmit-only capability (eg CANbus to some CANbus devices), or bidirectional, eg to other computer systems using a local area network or a wide area digital network. Several protocols and protocol stacks may be used on each of these networks and the network interfaces described above.

[0226] The aforementioned human interface devices, human accessible storage devices, and network interfaces can be connected to the core (2840) of the computer system (2800).

[0227] The core (2840) may include one or more central processing units (CPUs) (2841), graphics processing units (GPUs) (2842), specialized programmable processing modules in the form of Field Programmable Gate Arrays (FPGAs). ) (2843), hardware accelerators

(2844) for some tasks, etc. These devices, together with read-only memory (ROM) (2845), random access memory (2846), high-capacity internal storage such as internal hard drives (2847) not accessible to the user, SSD, etc., can be connected via the system bus (2848). In some computer systems, the system bus (2848) may be available in the form of one or more physical connectors to allow for expansions with additional CPUs, GPUs, and the like. Peripherals can be connected either directly to the system bus (2848) of the kernel, or via a peripheral bus (2849). Peripheral bus architectures include PCI, USB, and the like.

[0228] CPUs (2841), GPUs (2842), FPGAs (2843), and accelerators (2844) may execute some instructions that together may constitute the aforementioned computer code. This computer code may be stored in ROM (2845) or RAM (2846). Transient data may also be stored in RAM (2446), while persistent data may be stored, for example, in an internal mass storage (2847). Fast storage and retrieval from any storage device can be achieved through the use of cache memory, which can be closely associated with one or more CPU (2841), GPU (2842), mass storage (2847), ROM (2845), RAM (2846) etc.

[0229] Computer-readable media may store computer code for performing various computer-implemented operations. The media and computer code may be specially created for the purposes of the present invention or may be well known and available to those skilled in the art of computer software.

[0230] By way of example, and not limitation, a computer system having an architecture (2800), and in particular a core (2840), can provide functionality through execution by processor(s) (including(and) CPU, GPU, FPGAs, accelerators, etc.) software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with the user-accessible mass storage described above, as well as some persistent storage in the kernel (2840), such as internal mass storage (2847) or ROM (2845). Software implementing various embodiments of the present invention may be stored on such devices and executed by the core (2840). The computer-readable medium may include one or more memory devices or chips, in accordance with specific needs. The software may cause the core (2840) and in particular its processors (including CPUs, GPUs, FPGAs, etc.) to execute specific processes or specific parts of the specific processes described herein, including setting data structures stored in RAM (2846) , and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality through logic hardwired or otherwise embodied in circuitry (e.g., an accelerator (2844)) that may operate in place of or in conjunction with software to perform specific processes or specific portions of the specific processes described herein. A reference to software may cover logic, and vice versa when appropriate. Reference to computer-readable media may encompass a circuit (eg, an integrated circuit) that stores software to execute, a circuit that embodies execution logic, or both, as appropriate. The present invention encompasses any suitable combination of hardware and software.

Appendix A: Abbreviations

ASIC: Application Specific Integrated Circuit

BMS: a set of standards

CANBus: controller network bus

CBF: Coded Block Flag

CD: CD

CPU: central processing unit

CRT: kinescope

CTB: coding tree block

CTU: coding tree unit

CU: coding unit

DVD: Digital Video Disc

FPGA: Field Programmable Gate Array

GOP: group of images

GPU: GPUs

GSM: Global System for Mobile Communications

HEVC: High Performance Video Coding

HRD: Hypothetical Reference Decoder

ISP: intra-subdivision

IC: integrated circuit

JEM: collaborative research model

LAN: local area network

LCD: liquid crystal display

LTE: a project for the long-term development of communication systems

MRM: most probable mode

MV: motion vector

OLED: organic light emitting diode

RW: prediction block

PCI: Peripheral Component Interconnect

PLD: Programmable Logic Device

PU: prediction unit

RAM: random access memory

ROM: Read Only Memory

SBT: sub-block conversion

SEI: Additional Improvement Information

SNR: signal to noise ratio

SSD: solid state drive

TU: conversion unit,

USB: Universal Serial Bus

VPDU: virtual pipeline data unit

VUI: video suitability information

VVC: Universal Video Coding

[0231] Although several illustrative embodiments are described here, variations, permutations, and various replacement equivalents are possible and are within the scope of the invention. Thus, those skilled in the art can provide numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the invention and thus fall within its spirit and scope.

Claims (39)

1. A video decoding method in a decoder, in which: decoding encoded coding block (CB) information in an image from the encoded video bitstream, the encoded information indicating a CB width of W samples and an CB height of H samples; splitting the CB into processing subunits (SPUs) having a width and height, wherein the SPU width is equal to the minimum value of W and K, or the SPU height is equal to the minimum value of H and K, and at least one of the width W and the height H for CB larger than size K of the processing data unit; determining a partition structure for the additional partition of the SPU based on the width and height of the SPU and a maximum transform unit (TU) size of M samples, wherein at least one of the width and height of the SPU is greater than M; and partitioning each of the SPUs into M×M TUs based on a partitioning structure that has been determined. 2. The method according to claim 1, in which the width and height of the SPU is greater than M; when determining the partition structure, defining the partition structure as a partition structure in the form of a quadtree; and when splitting the SPU into TUs, split the SPU into TUs based on a quadtree split structure. 3. The method according to claim 1, in which the width of the SPU is greater than M and the height of the SPU is equal to M; when determining the partitioning structure, defining the partitioning structure as a vertical partitioning structure in the form of a binary tree; and when splitting the SPU into TUs, the SPUs are split into TUs based on the vertical splitting structure in the form of a binary tree. 4. The method according to claim 1, in which the height of the SPU is greater than M and the width of the SPU is equal to M; when determining the splitting structure, defining the splitting structure as a horizontal splitting structure in the form of a binary tree; and when splitting the SPU into TUs, split the SPU into TUs based on the horizontal split structure as a binary tree. 5. The method according to any one of paragraphs. 1-4, in which, when partitioning an SPU into TUs, one of the SPUs into TUs is recursively partitioned based on a partition structure. 6. The method according to any one of paragraphs. 1-4, further comprising: processing the SPU according to the first scan order; and processing the TUs in each of the SPUs according to the second scan order. 7. The method of claim 6, wherein at least one of the first scan order and the second scan order is one of (i) raster scan order, (ii) vertical scan order, (iii) zigzag order, and (iv) diagonal scan order scanning. 8. The method of claim 7, wherein the first scan order and the second scan order are raster scan orders. 9. The method according to p. 2, in which additionally: processing the SPU according to the first order of the scan; and processing the TUs in each of the SPUs according to the second scan order, wherein W is 128, H is 64, K is 64, and M is 32; the first scan order is from left to right; and the second scan order is the raster scan order. 10. The method according to any one of paragraphs. 1-4, in which the processing data unit size K indicates the size of the virtual pipeline data unit (VPDU); the first of the SPUs is included in the first VPDU and the second of the SPUs is included in the second VPDU in the picture; and additionally: after processing the first VPDU at the first stage of the multi-stage pipeline, simultaneously processing the first VPDU at the second stage of the multi-stage pipeline and the second VPDU at the first stage of the multi-stage pipeline. 11. The method of claim 1, wherein at least two TUs in one of the SPUs are configured to simultaneously process on different stages of a multi-stage pipeline. 12. The method according to claim 1, in which the partition of the CB contains: selecting a partition structure from a plurality of partition structures for partitioning the CB based on comparing at least one of the width W and the height H of the CB block with a maximum transformation unit (TU) size M. 13. The method of claim 12, wherein the plurality of partition structures comprise at least one of a quadtree partition structure, a vertical binary tree partition structure, and a horizontal binary tree partition structure, wherein: if the width W is greater than M and the width W is greater than the height H, selecting a vertical splitting structure in the form of a binary tree to split the CB into two parts, each of which has a size of W/2×H; if the height H is greater than M and the height H is greater than the width W, a horizontal splitting structure in the form of a binary tree is selected to split the CB into two parts, each of which has a size of W×H/2. 14. Device for video decoding, containing a processing circuit configured to implement the method according to any one of paragraphs. 1-13.

Images