RU2782437C1 - Colour conversion for video encoding - Google Patents

Abstract

FIELD: encoding.

SUBSTANCE: invention relates to video encoding tools. A syntax element indicating whether the residual blocks of the current coding unit (CU) have been processed using colour space conversion is received from the bitstream of encoded video data. The residual blocks of the current CU herein include a residual brightness block, a first residual chromaticity block, and a second residual chromaticity block. In response to the syntax element indicating that the residual blocks of the current CU have been processed using colour space conversion, based on the fact that the conversion skip flag corresponding to each of the residual blocks of the current CU equals 1 and the quantisation parameter (QP) corresponding to each of the residual blocks of the current CU equals 4, the colour space undergoes inverse conversion using the first variant of the colour space conversion equations to the residual blocks of the current CU in order to generate modified versions of the residual blocks of the current CU.

EFFECT: increased efficiency of video encoding.

12 cl, 30 dwg

Description

[0001] This application claims priority under U.S. Patent Application No. 17/035,218 entitled "Color Transformation for Video Coding" filed September 28, 2020, which claims priority under U.S. Provisional Patent Application No. 62/913,486 "Improved Color Transformation in VVC filed October 10, 2019.

FIELD OF TECHNOLOGY

[0002] Within the framework of the disclosure of the present invention describes embodiments generally related to video coding.

BACKGROUND OF THE INVENTION

[0003] The description of the background of the invention in this section is generally intended to provide a context for the disclosure of the present invention. The work of the inventors, to the extent that it is described in this section on the background of the invention, as well as aspects of the description that cannot be qualified as prior art at the date of filing of the application, are not considered, either directly or indirectly, as prior art. technique in relation to the disclosure of the present invention.

[0004] Encoding and decoding of video signals may be performed using inter prediction with motion compensation. The uncompressed digital video data may comprise a number of images, each of which has a spatial size, such as 1920x1080 luma samples and associated chrominance samples. The set of images may be characterized by a fixed or variable image rate (also informally referred to as frame rate), such as 60 images per second or 60 Hz. Uncompressed video data has significant bit rate requirements. For example, a 1080p60 4:2:0 video signal at 8 bits per sample (1920x1080 sample resolution at 60Hz frame rate) requires bandwidth close to 1.5 Gbps. An hour of such video takes up over 600 GB of storage space.

[0005] One of the goals of video encoding and decoding may be to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or reduce memory space requirements in some cases by two orders of magnitude or more. For this, compression can be used both lossless and lossy, as well as a combination of these methods. Lossless compression refers to technologies that can be used to restore an exact copy of the original signal from a compressed source signal. When lossy compression is used, the reconstructed signal may differ from the original signal, but the distortion between the original and reconstructed signals is so small that the reconstructed signal is suitable for the intended application. In the case of video, lossy compression is widely used. The acceptable level of distortion depends on the application; for example, users of certain consumer streaming applications may tolerate higher levels of distortion than users of TV distribution applications. The achievable compression ratio may reflect the fact that a higher allowable/acceptable level of distortion may allow higher compression ratios to be used.

[0006] The video encoder and video decoder may employ technologies from several broad categories including, for example, motion compensation, transformation, quantization, and entropy coding.

[0007] Technologies used in video codecs may include technologies known as intra-coding. With internal coding, sample values are presented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, the image is spatially divided into blocks of samples. If the blocks of samples are encoded in intra-mode, then the picture may be an intra-picture. The intra pictures and their derivatives, such as independent decoder update pictures, may be used to reset the decoder state and may thus be used as the first picture in the encoded video bitstream and video session, or as a still picture. The inner block samples may be transformed and the transform coefficients may be quantized before entropy encoding. Intra-coding may be a technique that minimizes sample values in the pre-transform region. In some cases, the smaller the DC value after transformation and the smaller the AC coefficients, the fewer bits are required for a given quantization step size to represent a block after entropy encoding.

[0008] Conventional intra coding, such as known from MPEG-2 generation coding technologies, does not use intra prediction. However, some more recent video compression techniques include methods that check blocks of data from, for example, ambient sample data and/or metadata obtained during encoding/decoding in the spatial environment and in the preceding decoding order. Such technologies are hereinafter referred to as "internal prediction" technologies. It should be noted that, in at least some cases, intra prediction only uses reference data from the current picture in the process of restoration, and not from the reference pictures.

[0009] There are many different forms of intra prediction. If several such methods can be used in a given video coding technology, then the method used can be encoded in intra prediction mode. In some cases, modes may have sub-modes and/or parameters that may be encoded individually or included in the mode codeword. The codeword to be used for a given mode/submode/parameter combination may affect the coding gain obtained using intra prediction, and thus may affect the entropy coding technology used to translate codewords into a bitstream.

[0010] Certain intra-prediction modes have been introduced in H.264 recommendations, enhanced in H.265, and further enhanced in more recent technologies such as Joint Exploration Model (JEM), Versatile Video Coding (VVC) and a set of benchmarks (BMS, BenchMark Set). The predictor block can be formed using the values of the surrounding samples belonging to already available samples. The surrounding sample values are copied into the predictor block according to the direction. A reference to the direction to be used may be encoded in the bitstream or directly predicted.

[0011] FIG. 1A, a subset of nine predictor directions out of 33 possible H.265 predictor directions (corresponding to 33 angle modes out of 35 internal modes) is shown in the lower right corner. The point where the arrows meet (101) represents the predictable reading. The arrows represent the direction from which the reading is predicted. For example, an arrow (102) indicates that the reading (101) is predicted based on the reading or readings in the upper right direction at an angle of 45 degrees with respect to the horizontal. Similarly, the arrow (103) indicates that the reading (101) is predicted based on the reading or readings in the lower left direction of the reading (101) at an angle of 22.5 degrees with respect to the horizontal.

[0012] FIG. 1A in the upper left corner shows a square block (104) with a dimension of 4x4 samples (marked with a dotted, bold line). The square block (104) contains 16 samples, each of which is marked with the letter "S", and is characterized by its position along the Y axis (eg, row index) and its position along the X axis (eg, column index). For example, sample S21 is the second sample on the Y-axis (top) and the first sample (left) on the X-axis. Similarly, sample S44 is the fourth sample in block (104) both on the Y-axis and on the X-axis. 4x4 countdown, S44 is in the bottom right corner. In addition, reference readings are shown that use a similar numbering scheme. The reference reference is denoted by the symbol R, its position along the Y axis (eg, row index) and X position (column index) relative to the block (104). According to H.264 and H.265, the prediction samples are adjacent to the block being reconstructed; thus, it is not required to use negative values.

[0013] Intra prediction may work by copying reference sample values from neighboring samples according to the reported prediction direction. For example, suppose the encoded bitstream contains signaling that for that block indicates a prediction direction consistent with arrow (102), then samples are predicted from the prediction sample or samples in the upper right direction at a 45 degree angle to the horizontal. In this case, samples S41, S32, S23 and S14 are predicted based on the same reference sample R05. Then, the sample S44 is predicted based on the reference sample R08.

[0014] In certain cases, the values of the plurality of reference samples may be combined, for example, by interpolation, in order to calculate the reference sample; especially if the directions are not evenly divisible by 45 degrees.

[0015] The number of possible directions increases as video coding technology advances. H.264 (2003) could present nine directions. The number of destinations has increased to 33 in H.265 (2013), and at the time of disclosure of the present invention, JEM/VVC/BMS technologies can support up to 65 destinations. Experiments have been carried out to identify the most likely directions, and certain entropy coding techniques have been used to represent these likely directions with a small number of bits, subject to a certain penalty for the less likely directions. In addition, the directions themselves may sometimes be predicted based on adjacent directions used in adjacent, already decoded blocks.

[0016] The intra prediction modes used in HEVC are shown in FIG. 1B. In HEVC, there are a total of 35 intra prediction modes, of which mode 10 is horizontal mode, mode 26 is vertical mode, and mode 2, mode 18 and mode 34 are diagonal modes. The intra prediction modes are communicated by the three most likely modes (MPM, Most Probable Mode) and the 32 remaining modes.

[0017] FIG. 1C shows intra prediction modes used in VVC. In HEVC, there are 95 intra prediction modes in total, as shown in FIG. 1C, of which mode 18 is a horizontal mode, mode 50 is a vertical mode, and mode 2, mode 34, and mode 66 are diagonal modes. Modes -1 ~ -14 and modes 67 ~ 80 are called Wide-Angle Intra Prediction (WAIP) modes.

[0018] The bit mapping of intra prediction directions in the encoded video bitstream that represents the direction may differ depending on the video coding technology, and may range from, for example, simple direct prediction direction transformation to codeword intra prediction mode to complex adaptive schemes that involve MRM and similar technologies. However, in all cases, there may be certain directions that are statistically less likely to occur in video content than other defined directions. Since the goal of video compression is to reduce redundancy, those least likely directions will be represented by more bits in a well-functioning video coding technology than more likely directions.

[0019] Encoding and decoding of video signals may be performed using inter prediction with motion compensation. The uncompressed digital video data may comprise a number of images, each characterized by a spatial size, such as 1920 x 1080 luma samples and associated chrominance samples. A set of images may be characterized by a fixed or variable image rate (also informally referred to as frame rate), for example. 60 images per second or 60 Hz. Uncompressed video data has significant bit rate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920x1080 sample resolution at 60Hz frame rate) requires bandwidth close to 1.5 Gbps. An hour of such video takes up more than 600 GB of storage space.

[0020] Motion compensation may be a lossy compression technology and may refer to technologies in which a block of sample data from a previously reconstructed image or a portion thereof (reference image) after being spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV, Motion Vector) is used to predict the newly reconstructed image or part of it. In some cases, the reference image may be the same as the current image that is being restored. The MV vectors have two dimensions, X and Y, or three dimensions, with the third being an indication of the reference picture used (the latter may indirectly be a temporal dimension).

[0021] According to some video data compression techniques, the MV applicable to a particular sample data region can be predicted based on other MVs, such as those associated with another sample data region spatially adjacent to the reconstructed region, and the preceding MV in decoding order. These actions can substantially reduce the amount of data required to encode the MV, thereby eliminating redundancy and increasing the compression ratio. MV prediction can work effectively, for example, because when encoding a video input coming from a camera (known as natural video), there is a statistical chance that areas that are larger than the area to which a particular MV is applied move in a similar direction and thus may in some cases be predicted using a similar motion vector derived from the MV vectors of the neighboring area. This has the effect that the MV found for a given area must be the same or the same as the MV predicted from the surrounding MVs, and in turn can be represented after entropy encoding with fewer bits than would be possible using direct MV encoding. In some cases, MV prediction may be an example of lossless compression of a signal (namely, MV vectors) derived from the original signal (namely, a stream of samples). In other cases, MV prediction may be directly lossy, for example, due to rounding errors when calculating a predictor based on multiple surrounding MVs.

[0022] Various MV prediction mechanisms are described in H.265/HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding", December 2016). Of the many mechanisms proposed in H.265 Recommendation, the following describes a technology called "spatial merge" (spatial merge).

[0023] FIG. 1D shows that the current block (110) contains the samples found by the encoder during the motion search process, predicted based on a previous block of the same size that was shifted in space. Instead of directly encoding the MV vector can be derived from the metadata associated with one or more reference pictures, for example, from the most recent (in decoding order) reference picture, using the MV associated with any of the five surrounding samples, denoted as A0, A1 and B0, B1, B2 (102 106, respectively). In H.265, the MV prediction process may use predictors from the same reference picture as used by an adjacent block. The order of forming a suitable list can be as follows: A0 → B0 → B1 → A1 → B2.

SUMMARY OF THE INVENTION

[0024] Within aspects of the disclosure of the present invention, a video decoding method performed by a video decoder is provided. The syntax element may be received from the encoded video bitstream. The syntax element may indicate that the residual blocks of the current coding unit (CU, Coding Unit) are processed using color space conversion. The residual blocks of the current CU may include a luma residual block, a first chrominance residual block, and a second chrominance residual block. In response to receiving a syntax element indicating that the residual blocks of the current CU are processed using a color space transform, two variants of the color space transform equations can be selected. An inverse color space transform using a chosen variant of the color space transform equations may be applied to the remaining blocks of the current CU.

[0025] According to an embodiment, the two versions of the color space conversion equations include a first version of the color space conversion equations, comprising the following equations:

tmp=rY[x][y] (rCb[x][y]>>1),

rY[x][y]=tmp+rCb[x][y],

rCb[x][y]=tmp-(rCr[x][y]>>1), and

rCr[x][y]=rCb[x][y]+rCr[x][y],

and the second version of the color space conversion equations, containing the following equations:

tmp=rY[x][y] - rCb[x][y],

rY[x][y]=rY[x][y]+rCb[x][y],

rCb[x][y]=tmp-rCr[x][y], and

rCr[x][y]=tmp+rCr[x][y].

The input data for each version of the color space transformation equations include an array of residual luminance samples of the residual luminance block containing the elements rY[x][y], an array of residual chrominance samples of the first residual chrominance block containing elements rCb[x][y], and an array of residual chromaticity samples of the second residual chrominance block containing elements rSk[x][y]. The output data for each version of the color space transformation equations includes a modified array of residual luminance samples of the residual luminance block containing the elements rY[x][y], a modified array of residual chrominance samples of the first residual chrominance block containing the elements rCb[x][y], and modified array of residual chrominance samples of the second residual chrominance block containing elements rCr[x][y].

[0026] According to an embodiment, the first version of the color space transform equations is selected if the skip transform flag corresponding to each of the residual blocks of the current CU is 1. According to the embodiment, the second version of the color space transform equations is selected if at least one of the skip transform flags corresponding to the residual blocks of the current CU is 0.

[0027] According to an embodiment, the first version of the color space transform equations is selected if the skip transform flag corresponding to each of the current CU's residual blocks is 1 and the Quantization Parameter (QP) corresponding to each of the current CU's residual blocks is 4. According to an embodiment, the second variant of the color space transformation equations is selected if at least one of the QPs corresponding to the residual blocks of the current CU is not equal to 4.

[0028] According to an embodiment, the received syntax element indicates that only the first version of the color space conversion equations is applied. A syntax element is one of the following: a slice level syntax element, an image level syntax element, or a sequence level syntax element. According to an embodiment, the received syntax element indicates that only the first variant of the color space transformation equations are applied, regardless of the value of a flag indicating whether transformation and quantization are skipped for the residual CUs. A syntax element is one of the following: a slice level syntax element, an image level syntax element, or a sequence level syntax element.

[0029] According to an embodiment, one of the two color space conversion equations is selected according to the value of the skip conversion flag corresponding to each of the residual blocks of the current CU. According to an embodiment, one of the two color space conversion equations is selected according to a variable indicating whether block-based Differential Pulse Code Modulation (BDPCM) is applied to the current CU.

[0030] According to an embodiment, the method may also include determining that a color space transform is not applied to the residual CUs if the CU's luma sample bit depth differs from the CU's chrominance sample bit depth. According to an embodiment, the method may also include determining that a color space transform is not applied to the residual CUs if the CU's luminance sample bit depth differs from the CU's chroma sample bit depth and the transform skip flag corresponding to each of the residual CUs is 1 or 0. In an embodiment, the method may also include determining that a color space transform is not applied to the residual CUs if the CU's luma bit depth differs from the CU's chroma sample bit depth, the QP corresponding to each of the residual CUs is 4, and a flag the transformation skip corresponding to each of the residual CUs is 1 or 0.

[0031] Within aspects of the disclosure of the present invention, an apparatus for video decoding is provided. The device may comprise a circuit configured to receive a syntax element from the encoded video bitstream indicating that the residual blocks of the current CU have been processed using a color space transform. The residual blocks of the current CU may include a luma residual block, a first chrominance residual block, and a second chrominance residual block. In response to receiving a syntax element indicating that the residual blocks of the current CU are processed using a color space transform, two variants of the color space transform equations can be selected. An inverse color space transform using a chosen variant of the color space transform equations may be applied to the remaining blocks of the current CU.

[0032] Within aspects of the disclosure, a computer-readable medium is provided that stores instructions that, when executed by a processor, perform a video decoding method.

BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Additional features, features and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which:

[0034] in FIG. 1A schematically shows an example of a subset of intra prediction modes.

[0035] FIG. 1B shows an example of intra prediction directions.

[0036] FIG. 1C shows an example of intra prediction directions.

[0037] FIG. 1D shows, as one example, the current block and surrounding spatial blocks suitable for merging.

[0038] FIG. 2 is a schematic representation of a simplified block diagram of a communication system in accordance with an embodiment of the present invention.

[0039] FIG. 3 is a schematic representation of a simplified block diagram of a communication system in accordance with an embodiment of the present invention.

[0040] FIG. 4 is a schematic representation of a simplified block diagram of a decoder according to an embodiment of the present invention.

[0041] In FIG. 5 is a schematic representation of a simplified block diagram of an encoder in accordance with an embodiment of the present invention.

[0042] FIG. 6 shows a block diagram of an encoder in accordance with another embodiment of the present invention.

[0043] FIG. 7 is a block diagram of a decoder according to another embodiment of the present invention.

[0044] FIG. 8A shows blocking in accordance with an embodiment of the present invention.

[0045] FIG. 8B shows a blocking tree in accordance with an embodiment of the present invention.

[0046] FIG. 9A shows partitioning using a vertical center-side ternary tree in accordance with an embodiment of the present invention.

[0047] FIG. 9B shows partitioning using a horizontal center-side ternary tree in accordance with an embodiment of the present invention.

[0048] FIG. 10A-10D show various color formats in accordance with embodiments of the present invention.

[00049] FIG. 11 shows an example of an encoder according to an embodiment of the present invention.

[0050] FIG. 12 shows an example of a decoder according to an embodiment of the present invention.

[0051] FIG. 13 shows a straight line between the minimum and maximum brightness values according to an embodiment of the present invention.

[0052] FIG. 14A and 14B show the sample locations used to derive α and β in LT_CCLM in accordance with an embodiment of the present invention.

[0053] FIG. 15A and 15B show the sample locations used to derive α and β in T_CCLM in accordance with an embodiment of the present invention.

[0054] FIG. 16A and 16B show sample locations used to derive α and β in L_CCLM in accordance with an embodiment of the present invention.

[0055] FIG. 17 shows an example of dividing adjacent samples into two groups in accordance with an embodiment of the present invention.

[0056] FIG. 18 is a schematic representation of an encoder and decoder according to an embodiment of the present invention.

[0057] FIG. 19 shows an embodiment of a process performed by a decoder.

[0058] FIG. 20 is a schematic representation of a computer system in accordance with an embodiment of the disclosure of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

[0059] I. Video encoder and video decoder system

[0060] FIG. 2 shows a simplified block diagram of a communications system (200) in accordance with an embodiment of the disclosure of the present invention. The communication system (200) includes a plurality of terminal devices that can communicate with each other, for example, via a network (250). For example, the communication system (200) comprises a first pair of terminal devices (210) and (220) interconnected via a network (250). In FIG. 2, for example, the first pair of terminal devices (210) and (220) perform unidirectional data transmission. For example, a terminal device (210) may encode video data (eg, a video stream captured by a terminal device (210)) for transmission to another terminal device (220) via a network (250). The encoded video data may be transmitted as one or more encoded video bitstreams. The terminal device (220) may receive the encoded video data from the network (250), decode the encoded video data to reconstruct the video images, and display the video images in accordance with the decoded video data. Unidirectional data transfer may be a common feature in media serving applications and the like.

[0061] In another example, the communication system (200) includes a second pair of terminal devices (230) and (240) performing two-way transmission of encoded video data, which can be carried out, for example, during a video conference. In this example, for two-way communication, each terminal device (230) and (240) may encode video data (for example, a video stream captured by the terminal device) for transmission to another terminal device of a pair of terminal devices (230) and (240) via a network (250 ). Each terminal device of the pair of terminal devices (230) and (240) may also receive the encoded video data transmitted by the other terminal device of the pair of terminal devices (230) and (240) and may decode the encoded video data to reconstruct the video images and display the video images on an available display device in according to the recovered video data.

[0062] In the example shown in FIG. 2, the terminal devices (210), (220), (230), and (240) may be servers, personal computers, and smartphones, however, the principles of the disclosure of the present invention may not be limited to these devices. In embodiments of the disclosure of the present invention, application is found for laptops, tablets, media players and/or specialized video conferencing equipment. Network (250) represents any number of networks that transmit encoded video data between terminal devices (210), (220), (230) and (240), including, for example, wired and/or wireless communication networks. The communications network (250) may communicate over circuit-switched or packet-switched links. Typical networks include telecommunications networks, local and wide area networks and/or the Internet. For the purposes of this discussion, it may be assumed that the architecture and topology of the network (250) is not essential to the principles of the disclosure of the invention, unless otherwise indicated below.

[0063] FIG. 3 shows the location of a video encoder and decoder in a streaming environment as an example of the application of the disclosed subject matter. The disclosed subject matter of the present invention may equally be used with other valid video applications including, for example, video conferencing, digital television, storage of compressed video data on digital media including CD, DVD, memory card, and the like.

[0064] The streaming system may include a capture subsystem (313), which may include a video source (301), such as a digital camera, that creates, for example, an uncompressed video stream (302). For example, the video stream (302) contains samples that are taken by a digital camera. The stream (302), shown in bold to draw attention to the large amount of data compared to the encoded video data (304) (or encoded video bitstreams), may be processed by an electronic device (320) that contains a video encoder (303) connected to the source ( 301) of the video signal. Video encoder (303) may include hardware, software, or a combination of these components to enable or implement aspects of the subject disclosure of the present invention in accordance with the more detailed description below. Encoded video data (304) (or encoded video bitstream (304)), depicted as a thin line to draw attention to the small amount of data compared to the video stream (302), may be stored on the streaming server (305) for further use. One or more streaming client subsystems, such as the client subsystems (306) and (308) shown in FIG. 3, can access the streaming server (305) to retrieve copies (307) and (309) of the encoded video data (304). The client subsystem (306) may contain a video decoder (310), for example, in an electronic device (330). The video decoder (310) decodes the incoming copy (307) of the encoded video data and creates an outgoing video stream (307) that can be rendered on the display (312) (for example, on a display screen) or on another visualization device (not shown). In some streaming systems, encoded video data (304), (307), and (309) (eg, video bitstreams) may be encoded according to certain video coding/compression standards. An example of such standards would be ITU-T Recommendation H.265. For example, a video data coding standard informally known as Versatile Video Coding (VVC) is under development. The disclosed subject matter of the present invention may be used in the context of VVC.

[0065] It should be noted that the electronic devices (320) and (330) may contain other components (not shown in the drawing). For example, the electronic device (320) may include a video decoder (not shown), and the electronic device (330) may also include a video encoder (not shown).

[0066] Figure 4 shows a block diagram of a video decoder 410 in accordance with an embodiment of the disclosure of the present invention. The video decoder (410) may be built into the electronic device (430). The electronic device (430) may include a receiver (431) (eg, receiver circuitry). Video decoder (410) may be used instead of video decoder (310) shown in the example of FIG. 3.

[0067] In this and other embodiments, the receiver (431) may receive one or more encoded video sequences to be decoded by the decoder (410), one sequence at a time, with the decoding of each encoded video sequence independent of other encoded video sequences. The encoded video sequence may be received over a channel (401), which may be a hardware/software link to a storage device storing the encoded video data. The receiver (431) may receive encoded video data along with other data, such as encoded audio data and/or service data streams, which may be sent to their respective application entities (not shown). The receiver (431) may separate the encoded video sequence from other data. A buffer memory (415) can be installed between the receiver (431) and the entropy decoder/analyzer (420) (hereinafter referred to as "analyzer" (420)) to eliminate network jitter. In certain applications, the buffer memory (415) is part of the video decoder (410). In other applications, the memory may be located outside the video decoder (410) (not indicated in the drawing). In some other applications, there may be a buffer memory (not indicated in the drawing) outside the video decoder (410), for example, to eliminate network jitter, and additionally another buffer memory (415) in the video decoder (410), for example, to process playback time intervals. If the receiver (431) receives data from a storage/forwarder with sufficient bandwidth and manageability, or from an isosynchronous network, the buffer memory (415) may not be used or may be small. For use in best quality of service packet networks such as the Internet, buffer memory (415) may be required, which may be relatively large and may advantageously adapt in size, and at least in part, this memory may be implemented in the operating system or in similar elements (not indicated in the drawing) outside the video decoder (410).

[0068] The video decoder (410) may include an analyzer (420) for recovering symbols (421) from the encoded video sequence. Categories of such symbols include, for example, information used to control the operation of the video decoder (410), as well as potential information to control a visualization device such as a visualization device (412) (for example, a display) that is not integrated with the electronic device (430), but may be associated with an electronic device (430) as shown in FIG. 4. Control information for the imaging device(s) may be transmitted in the format of Supplementary Enhancement Information (SEI) messages or Video User Friendly Information (VUI) parameter set fragments. , Video Usability Information) (not shown in the drawing). The analyzer (420) may analyze/entropy decode the received video sequence. Encoding of the encoded video sequence may be performed in accordance with a video coding technology or standard, and may follow various principles including variable length coding, Huffman coding, context sensitive or context sensitive arithmetic coding, and so on. The analyzer (420) can extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one of the parameters corresponding to the group. Subgroups can include groups of pictures (GOP, Groups of Pictures), images, tiles, slices, macroblocks, coding units (CU, Coding Unit), transformation units (TU, Transform Unit), prediction units (PU, Prediction Unit), etc. d. The analyzer (420) can also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0069] The analyzer (420) may perform an entropy decoding/analysis operation of the video sequence received from the buffer memory (415) to create symbols (421).

[0070] Many different blocks may be involved in symbol recovery (421), depending on the type of encoded video image or part thereof (eg, intra-coded image, outer or inner block) and other factors. The blocks involved and how they are used can be controlled using subgroup control information, which is analyzed based on the encoded video sequence by the analyzer (420). The flow of such subgroup control information between the analyzer (420) and the plurality of blocks is not further described below for simplicity.

[0071] In addition to the functional blocks already mentioned, the video decoder (410), as described below, can be conceptually subdivided into a number of other functional blocks. In practical implementations, when operating under commercial constraints, many of these blocks interact closely with each other and can be at least partially integrated into each other. However, for the purpose of describing the disclosed subject matter, the following conceptual division into functional blocks applies.

[0072] The first block is the scaling/inverse transform block (451). The scaling/inverse transform block (451) receives the quantized transform coefficient, as well as control information, including information about the transform used, block size, quantization factor, quantization scaling matrices, etc., in the form of symbol(s) (421) coming from the analyzer (420). Block (451) scaling /inverse transformation may output blocks containing sample values that can be fed to the aggregator (455).

[0073] In some cases, the output samples of the scaling/inverse transform block (451) may belong to an intra-coded block; that is, a block that does not use prediction information from previously restored images, but can use prediction information from previously restored portions of the current image. Such predictive information may be provided by intra prediction block 452. In some cases, the block (452) intra prediction generates a block of the same size and shape as the restored block, using the surrounding already restored information extracted from the buffer (458) current images. The current picture buffer (458) buffers, for example, a partially restored current picture and/or a fully restored current picture. The aggregator (455) in some cases adds, based on the samples, the prediction information generated by the intra prediction block (452) to the sample output provided by the scaling/inverse transform block (451).

[0074] In other cases, the output samples of the scaling/inverse transform block (451) may belong to an inter-coded block, possibly with motion compensation. In such a case, the motion compensation predictor (453) may access the reference picture memory (457) to retrieve samples used for prediction. After motion compensation, the extracted samples according to the symbols (421) belonging to the block may be added by the aggregator (455) to the output data of the scaler/inverse transform block (451) (in this case, this data is called residual samples or residual signal), in order to generate output information containing samples. Reference picture memory addresses (457) from which motion compensation predictor (453) extracts prediction samples may be controlled by motion vectors available to motion compensation predictor (453) in the form of symbols (421), which may be characterized, for example, by X components , Y and a reference image. Motion compensation may also include interpolation of sample values when retrieving reference pictures from memory (457) if accurate reduced sample motion vectors, motion vector prediction mechanisms, etc. are used.

[0075] The output samples of the aggregator (455) can be processed using various ring filtering technologies in the loop filter block (456). The video data compression technologies may include parameter-driven in-loop filtering technologies that are included in an encoded video sequence (also referred to as a coded video bitstream) and made available to the in-loop filtering unit (456) as symbols (421) from the parser (420), but may also respond to meta-information obtained in the process of decoding the previous (in the order of decoding) parts of the encoded image or encoded video sequence, as well as responding to previously restored sample values subjected to contour filtering.

[0076] The output of the loop filter block (456) may be a stream of samples that may be fed to the renderer (412) as well as stored in the reference picture memory (457) for later use in inter prediction.

[0077] Certain encoded pictures after full recovery can be used as reference pictures for subsequent prediction. For example, after the encoded picture corresponding to the current picture is fully restored and identified as a reference picture (eg, by the analyzer (420)), the current picture buffer (458) may become part of the reference picture memory (457), and the updated current picture buffer may be reallocated before starting the reconstruction of the next encoded picture.

[0078] The video decoder (410) may perform decoding operations in accordance with a predetermined video data compression technology defined in a standard such as ITU-T recommendation H.265. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used, in the sense that the encoded video sequence adheres to the syntax of the video compression technology or standard and the profiles specified in the video compression technology or standard documentation. More specifically, a profile may select certain media as the only ones available for use under the control of a given profile from all available media defined by a video compression technology or standard. In addition, for compliance, the complexity of the encoded video sequences may be within the limits defined by the level of technology or video compression standard. In some cases, complexity levels limit the maximum image size, maximum frame rate, maximum recovery sample rate (measured in Ms/s, for example), maximum reference image size, and so on. Complexity levels may also be limited in some cases by Hypothetical Reference Decoder (HRD) specifications and metadata for managing the HRD buffer reported in the encoded video sequence.

[0079] According to an embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video information. Additional data may be inserted into the encoded video sequence(s). The additional data may be used by the video decoder (410) to correctly decode the data and/or to recover the original video data more accurately. The additional data may be provided in the form of, for example, temporal, spatial, or enhanced Signal Noise Ratio (SNR) levels, redundant slices, redundant images, forward error correction codes, and so on.

[0080] FIG. 5 shows a block diagram of a video encoder (503) in accordance with an embodiment of the disclosure of the present invention. The video encoder (503) is integrated into the electronic device (520). The electronic device (520) includes a transmitter (540) (eg, a transmission circuit). A video encoder (503) may be used in place of the video encoder (303) shown in the example of FIG. 3.

[0081] The video encoder (503) may receive video samples from a video signal source (501) (not part of the electronic device (520) shown in the example in FIG. 5) that may capture the video image(s) to be encoded by the video encoder (503 ). In another example, the video signal source (501) is part of the electronic device (520).

[0082] The video source (501) may provide the original video sequence to be encoded by the video encoder (503) as a digital video sample stream that may have any suitable bit depth (e.g., 8 bit, 10 bit, 12 bit, ...), any color space (for example, VT.601 Y CrCB, RGB, ...) and any suitable frame of references (for example, Y CrСb 4:2:0, Y CrСb 4:4:4). In the serving media system, the video signal source (501) may be a storage device that stores previously prepared video data. In a videoconferencing system, the video source (501) may be a camera that captures local graphics information in the form of a video sequence. The video data may be provided as a plurality of individual images that convey motion when viewed sequentially. The images themselves can be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc. used. It is not difficult for a person skilled in the art to understand the relationship between pixels and samples. In the following description, samples are considered.

[0083] According to an embodiment, the video encoder (503) may encode and compress images of the original video sequence to convert it to the encoded video sequence (543) in real time or subject to any other time constraints depending on the application. Providing a suitable code rate is one of the functions performed by the controller (550). The controller (550) may also control other functional blocks, as described below, and may be operatively connected to these functional blocks. For simplicity, this connection is not indicated in the drawing. Parameters set by the controller (550) may include parameters related to rate control (picture skip, quantizer, lambda value for rate distortion optimization technologies, ...), picture size, group of pictures (GOP) level, maximum motion vector search range etc. The controller (550) may be configured to perform other suitable functions that relate to the video encoder (503) optimized for a particular system structure.

[0084] In some embodiments, the video encoder (503) is configured to operate in an coding loop. In an oversimplified description, for example, an encoding loop may include a source encoder (530) (for example, responsible for creating symbols such as a symbol stream based on the input image to be encoded and the reference image(s)) and a (local) decoder ( 533) built into the video encoder (503). The decoder (533) decompresses the symbols to create the sample data in a similar manner to the (remote) decoder (because any compression between the symbols and the encoded video bitstream is lossless according to the video compression techniques discussed in the disclosure of the subject matter of the present invention). The reconstructed sample stream (sample data) is written to the reference picture memory 534 . Since character stream decoding leads to bit-accurate results regardless of the location of the decoder (local or remote), the reference picture memory (534) also contains exact bits between the local encoder and the remote encoder. In other words, the encoder's prediction subsystem "sees" as reference picture samples exactly the same sample values that the decoder would "see" when using prediction in the decoding process. This fundamental principle of reference picture synchronism (and the resulting shift if synchronism cannot be maintained due to, for example, channel errors) is also used in some relevant areas.

[0085] The operation of the "local" decoder (533) may be similar to the operation of the "remote" decoder, such as the video decoder (410), which has already been described in detail above with reference to Fig.4. However, as shown in FIG. 4, since the symbols are available and encoding/decoding of the symbols into the encoded video sequence by the entropy encoder (545) and parser 420 can be lossless, the entropy decoding subsystems of the video decoder (410), including the buffer memory (415) and parser ( 420) may not be fully implemented in the local decoder (533).

[0086] It should be noted here that any decoding technique other than analysis/entropy decoding used in a decoder is also necessarily required in a substantially identical functional form in a corresponding encoder. For this reason, in the disclosure of the subject matter of the present invention, emphasis is placed on the operation of the decoder. The description of the encoder technologies can be abbreviated as they are the inverse of the comprehensively described decoder technologies. Only certain areas require a more detailed description below.

[0087] In operation, according to some examples, the source encoder (530) may perform motion-compensated predictive coding in which an input picture is predictively encoded with reference to one or more previously encoded pictures of a video sequence, referred to as "reference pictures". Thus, the encoding engine (532) encodes the differences between the pixel blocks of the input image and the pixel blocks of the reference image(s) that can be selected as prediction reference(s) to the input image.

[0088] The local video decoder (533) may decode the encoded video data of pictures, which may be referred to as reference pictures, based on the symbols generated by the source encoder (530). The encoding core (532) operations may advantageously be lossy processes. If the encoded video data can be decoded in a video decoder (not shown in FIG. 5), then the reconstructed video sequence can typically be a copy of the original video sequence with some errors. The local video decoder (533) duplicates the decoding processes that may be performed by the reference picture video decoder and may store the reference pictures in the reference picture cache (534). Thus, the video encoder (503) can locally store copies of the reconstructed reference pictures having the same content as the recovered reference pictures to be received by the far-end video decoder (no transmission errors).

[0089] The predictor (535) may perform a predictive search for the encoding kernel (532). That is, for a new picture to be encoded, the predictor (535) may search for sample data in the reference picture memory (534) (as suitable reference pixel blocks) or certain metadata such as reference picture motion vectors, block shapes, etc. which can serve as a suitable prediction reference for new images. The predictor (535) may operate in a sample block-by-pixel block matching mode to find suitable prediction references. In some cases, determined by the search results obtained by the predictor (535), the input image may be characterized by prediction links obtained based on a plurality of reference pictures stored in the reference picture memory (534).

[0090] The controller (550) may control the encoding operations of the source encoder (530), including, for example, setting parameters and subgroup parameters used to encode video data.

[0091] The output of all the functional blocks mentioned above can be entropy encoded in an entropy encoder (545). An entropy encoder (545) translates symbols generated by various functional blocks into an encoded video sequence by lossless symbol compression according to techniques such as Huffman coding, variable length coding, arithmetic coding, and so on.

[0092] The transmitter (540) may buffer the encoded video sequence(s) created by the entropy encoder (545) in order to prepare it for transmission over a communication channel (560), which may be a hardware/software link to a storage device, in which the encoded video data is to be stored. The transmitter (540) may combine the encoded video data from the video encoder (503) with other data to be transmitted, such as encoded audio and/or service data streams (these sources are not shown).

[0093] The controller (550) may control the operation of the video encoder (503). During the encoding process, the controller (550) may assign to each encoded picture a certain type of encoded picture, which may influence the encoding technology that can be applied to the corresponding picture. For example, images can often be assigned one of the following types:

[0094] An intra picture (I picture) may be an picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs provide various types of internal images, including, for example, Independent Decoder Refresh (IDR) images. One of skill in the art is aware of these I-image variants, their uses and features.

[0095] A predictive picture (P-picture) can be encoded and decoded using intra or inter prediction with no more than one motion vector and a reference index to predict sample values of each block.

[0096] A bi-directional predictive picture (B-picture) can be encoded and decoded using intra or inter prediction with no more than two motion vectors and reference indices to predict sample values of each block. Similarly, multi-prediction pictures may use more than two reference pictures and associated metadata to reconstruct a single block.

[0097] Source images can generally be spatially divided into multiple sample blocks (eg, 4x4, 8x8, 4x8, or 16x16 sample blocks) and encoded in a block-by-block fashion. The blocks may be predictively encoded with reference to other (already encoded) blocks, as determined by the purpose of the encoding applied to the corresponding block images. For example, I-picture blocks may be encoded without prediction or predictively with reference to already encoded blocks of the same picture (spatial prediction or intra prediction). P-picture pixel blocks may be predictively encoded using spatial prediction or temporal prediction with reference to one pre-coded reference picture. Blocks of B-pictures may be predictively encoded using spatial prediction or temporal prediction with reference to one or two pre-coded reference pictures.

[0098] The video encoder (503) may perform encoding operations in accordance with a predetermined video coding technology or standard, such as ITU-T H.265 recommendation. In operation, the video encoder (503) may perform various compression operations, including predictive coding operations that exploit temporal or spatial redundancy in the input video sequence. The encoded video data may thus conform to the syntax specified by the video coding technology or standard used.

[0099] According to an embodiment, the transmitter (540) may transmit additional data along with the encoded video information. The source encoder (530) may include such data in the encoded video sequence. Additional data may include temporal/spatial/SNR enhancement levels, other types of redundant data such as redundant images and slices, SEI messages, VUI parameter set fragments, and so on.

[0100] The video data may be captured as a plurality of source images (video images) in time sequence. Intra-prediction mode (often abbreviated as intra-prediction) uses the spatial correlation in a given image, while inter-prediction mode uses (temporal or other) correlation between images. For example, a specific encoded/decoded picture, called the current picture, is divided into blocks. If a block in the current picture is similar to a reference block in a previous encoded and still buffered reference picture in the video data, then the block in the current picture may be encoded with a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension identifying the reference picture when using multiple reference pictures.

[0101] In some embodiments, dual prediction technology may be used in an inter prediction process. The dual prediction technique uses two reference pictures, such as a first reference picture and a second reference picture, each of which precedes in decoding order the current picture in the video sequence (but may be past or future, respectively, in display order). A block in the current picture may be encoded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. Block prediction may be performed using a combination of the first reference block and the second reference block.

[0102] In addition, in the inter prediction process, a combining mode can be used to improve coding efficiency.

[0103] In accordance with some embodiments of the disclosure of the present invention, inter and intra prediction are performed in units of blocks. For example, according to the HEVC standard, an image in a video sequence is divided into Coding Tree Units (CTUs) for compression, and the size of the CTUs in the image is the same, for example, 64x64 pixels, 32x32 pixels, or 16x16 pixels. In general, the CTU contains three Coding Tree Blocks (CTBs), one luma CTB and two chrominance CTBs. Each CTU may be recursively split into one or more coding units (CUs) in the quadtree. For example, a 64x64 pixel CTU may be divided into one 64x64 pixel CU, or 4 32x32 pixel CUs, or 16 16x16 pixel CUs. For example, each CU is analyzed to determine the type of prediction for the CU, such as inter or intra prediction. The CU is divided into one or more prediction units (PU, Prediction Unit) depending on temporal and/or spatial predictability. Typically, each PU contains a luma Prediction Block (PB) and two chrominance PBs. According to an embodiment, the encoding (encoding/decoding) prediction operation is performed in prediction units. When using a luminance prediction unit, for example, the prediction unit contains a matrix of values (eg, luminance values) for pixels, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, and the like.

[0104] In FIG. 6 is a block diagram of a video encoder (603) according to another embodiment of the present invention. The video encoder (603) is configured to receive a processing unit (eg, a prediction unit) of sample values within the current video image in the video sequence and encoding the processing unit into an encoded image that is part of the encoded video sequence. For example, a video encoder (603) is used in place of the video encoder (303) shown in the example of FIG.

[0105] In an example of the HEVC standard, the video encoder (603) receives a matrix of sample values for a processing block, such as an 8x8 processing block, or the like. The video encoder (603) determines whether the best way to encode the processing block is intra mode, inter mode, or dual prediction mode, using rate distortion optimization, for example. If the processing block is to be encoded in intra mode, the video encoder (603) may use an intra prediction technique to encode the processing block into an encoded picture; and if the processing block is to be encoded in inter mode or dual prediction mode, the video encoder (603) may use inter prediction or dual prediction, respectively, to encode the processing block into an encoded picture. According to certain video coding techniques, the combining mode may be an inter prediction sub-mode in which a motion vector is derived based on one or more motion vector predictors without a gain associated with a coded component of the motion vector outside of the predictors. In some other video coding technologies, there may be a motion vector component applicable to the block in question. For example, the video encoder (603) includes other components such as a mode decision module (not shown) for determining the mode of the processing units.

[0106] In the example shown in FIG. 6, the video encoder (603) is shown to comprise an inter-encoder (630), an intra-encoder (622), a residual signal calculator (623), a switch (626), a residual encoder (624), a general purpose controller (621), and an entropy encoder (625) connected to each other as shown in FIG. 6.

[0107] The inter-coder (630) is configured to receive samples of the current block (e.g., processing block), compare the block to one or more reference blocks in the reference pictures (e.g., blocks in previous pictures and next pictures), generate inter prediction information ( for example, descriptions of redundant information according to inter coding technology, motion vectors, combining mode information) and calculation of inter prediction results (eg, block predictor) based on inter prediction information using any suitable technology. In some examples, reference pictures are decoded reference pictures decoded based on encoded video information.

[0108] The intra-encoder (622) is configured to receive samples of the current block (e.g., processing block), in some cases to compare the block with blocks already encoded in the same picture, to generate post-transform quantization coefficients, and in some cases also to generating intra prediction information (eg, intra prediction direction information according to one or more intra coding technologies). According to an example, the intra encoder 622 also calculates intra prediction results (eg, predicted block) based on intra prediction information and reference blocks in the same picture.

[0109] The general purpose controller (621) is configured to determine the common control data and to control other components of the video encoder (603) based on the common control data. In the example, the general purpose controller (621) determines the mode of the block and sends a control signal to the switch (626) in accordance with the mode. For example, when operating in intra mode, the general purpose controller (621) controls a switch (626) to select an intra mode result for use by residual data calculator (623) and controls an entropy encoder (625) to select intra prediction information and include it in bitstream; and when operating in the external mode, the general purpose controller (621) controls the switch (626) to select the inter prediction result for use by the residual data calculator (623), and also controls the entropy encoder (625) to select the inter prediction information and include it in the bit flow.

[0110] The residual data calculator (623) is configured to calculate the difference (residual data) between the received block and the prediction results selected from the intra-coder (622) or inter-coder (630). The residual encoder (624) is configured to operate on the residual data to encode the residual data to generate transform coefficients. According to an example, the residual encoder (624) is configured to convert the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then processed by quantization to obtain quantized transform coefficients. In various embodiments, the video encoder (603) also includes a residual data decoder (628). The residual data decoder (628) is configured to perform inverse transformation and generate decoded residual data. The decoded residual data may be appropriately used by the intra encoder (622) and inter encoder (630). For example, the inter-coder (630) may generate decoded blocks based on the decoded residual data and inter-prediction information, and the intra-coder (622) may generate decoded blocks based on the decoded residual data and intra prediction information. The decoded blocks are suitably processed to generate decoded pictures that can be buffered in a memory circuit (not shown) and used as reference pictures in some examples.

[0111] The entropy encoder (625) is configured to format the bitstream to include a coded block. The entropy encoder (625) is configured to include various information according to a suitable standard such as the HEVC standard. For example, the entropy encoder (625) is configured to include common control data, selected prediction information (eg, intra or inter prediction information), residual information, and other suitable information in the bitstream. It should be noted that, in accordance with the disclosed subject matter of the present invention, there is no residual information if the block is encoded in the combining submode of either the inter mode or the dual prediction mode.

[0112] Figure 7 shows a block diagram of a video decoder (710) in accordance with another embodiment of the disclosure of the present invention. The video decoder (710) is configured to receive encoded pictures that are part of the encoded video sequence and decode the encoded pictures to generate reconstructed pictures. For example, a video decoder (710) is used in place of the video decoder (310) shown in the example of FIG. 3.

[0113] In the example shown in FIG. 7, the video decoder (710) is shown to include an entropy decoder (771), an inter decoder (780), a residual data decoder (773), a reconstruction unit (774), and an intra decoder (772) connected to each other as shown. in fig. 7.

[0114] The entropy decoder (771) may be configured to recover from the encoded images certain characters that represent the syntax elements that make up the encoded image. Such symbols may include, for example, the mode in which the block is encoded (such as, for example, intra mode, inter mode, dual prediction mode, the last two of which are in the merge submode or another submode), prediction information (such as such as intra-prediction or inter-prediction) that can identify a specific sample or metadata used for prediction performed by the intra-decoder (772) or inter-decoder (780), respectively, the residual information in the form of, for example, quantized transform coefficients and etc. For example, if the prediction mode is inter mode or dual prediction mode, then inter prediction information is provided to the inter decoder (780), and if the prediction mode is intra mode, then intra prediction information is provided to the intra decoder (772). The residual information may be processed by inverse quantization and provided to a residual data decoder (773).

[0115] The inter decoder (780) is configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0116] The intra decoder (772) is configured to receive intra prediction information and generate intra prediction results based on the intra prediction information.

[0117] The residual data decoder (773) is configured to perform inverse quantization to extract the non-quantized transform coefficients and process the non-quantized transform coefficients to convert the residual data from the frequency domain to the spatial domain. The residual data decoder (773) may also require certain control information (including a Quantizer Parameter (QP)), and this information may be represented by the entropy decoder (771) (data paths not shown as this may be only a small amount of control information). information).

[0118] The recovery module (774) is configured to combine in the spatial domain the residual data in the form of the decoder (773) output of the residual data and the prediction results (in the form of the output of the inter and intra prediction modules, as the case may be) to form a reconstructed block, which may be part of the reconstructed image, which in turn may be part of the reconstructed video data. It should be noted that other suitable operations such as a release operation and the like may be performed to improve rendering quality.

[0119] It should be noted that video encoders (303), (503) and (603) and video decoders (310), (410) and (710) may be implemented using any suitable technology. It should be noted that video encoders (303), (503) and (603) and video decoders (310), (410) and (710) may be implemented using one or more integrated circuits. According to another embodiment, video encoders (303), (503) and (503) and video decoders (310), (410) and (710) may be implemented using one or more processors executing program instructions.

[0120] II. Blocking based on a tree structure

[0121] In accordance with some embodiments, the CTU is divided into CUs using a Quad Tree Binary Tree (QTBT) structure, called a coding tree, to adapt to different local characteristics of the individual blocks included in the CU. The decision as to which prediction - extrinsic (temporal) or intrinsic (spatial) -should be applied when encoding an image area may be made at the CU level. Each CU may also be partitioned into one, two or four PUs according to the partition type of the PUs. In some embodiments, the same prediction process is applied within the same PU and the corresponding information is passed to a decoder based on the PU. After obtaining a residual block by applying a prediction process based on the partition type to the PU, the CU may be partitioned into TUs according to a different quadtree structure similar to the quadtree structure used for the coding tree for the CTU. In some embodiments, the PU contains only one TU having the same shape as the PU.

[0122] The coding tree for a CTU may include many partition types, including CUs, PUs, and TUs. In some embodiments, the CU or TU is only square in shape, while the PU may be square or rectangular in shape for an inter prediction block. In other embodiments, rectangular shaped CUs, PUs, and TUs are allowed. An implicit quadtree-based partitioning may be applied at the image boundary such that the block retains the quadtree-based partitioning as long as the size of the partitioned block matches the image boundary. In accordance with some embodiments, the implicit split means that the split flag is not signaled, but is implied. For example, implicit QT means that only QT splitting is allowed for the bounding box of an image. Thus, the split flag is not signaled at the border of the image. According to another example, if only BT separation is allowed at the image boundary, then the implicit separation is binary separation. In some embodiments, if both QT and BT are allowed at the edge of the image, there is no implicit split, and the split method is signaled explicitly.

[0123] In accordance with some embodiments, the QTBT structure does not include multiple partition types (eg, QTBT does not include CU, PU, and TU partitioning) and supports more flexibility for CU partitioning shapes. For example, in a QTBT block structure, the CU may be either square or rectangular. 8A shows an example of a CTU (800) that is broken up by a QTBT structure. For example, CTU (800) is divided into four equal-sized CUs: (A), (B), (C), and (D). In FIG. 8B shows the corresponding coding tree illustrating branches related to CUs: (A), (B), (C), and (D). The solid lines show the quadtree based partitioning, and the dotted lines show the binary tree based partitioning. The binary tree structure may include two types of partitioning: (i) symmetrical horizontal partitioning and (ii) symmetrical vertical partitioning. For each splitting node (i.e., not a leaf) of the binary tree, one flag may be reported indicating what type of splitting (eg, horizontal or vertical) is being used, with 0 indicating a horizontal split and 1 indicating a vertical split, or vice versa. For quadtree-based partitioning, the type of partitioning is not specified, since in this case both horizontal and vertical partitioning of the block is performed to form 4 sub-blocks of the same size.

[0124] As shown in FIG. 8A and 8B, the CU subblock (A) is first split into two subblocks by a vertical split, with the left subblock split again by another vertical split. Sub-block CU (B) is further split by horizontal splitting. The CU subblock (C) is further partitioned by another quadrant partition using partitioning. The upper left sub-block of the sub-CU (C) is split with a vertical split followed by a split with a horizontal split. In addition, the lower right sub-block of the sub-CU (C) is split using a horizontal split. The upper right and lower left sub-blocks of sub-block CU (C) are not split further. The CU(D) subblock is not split further and thus does not include any additional leaf nodes in the coding tree below branch "D".

[0125] The leaf nodes of the binary tree may be called CUs, wherein binary division may be used for prediction and transformation processing without any additional division, and this means that the sizes of CU, PU and TU are the same in the structure of QTBT coding units. A CU may contain coding units (CBs) of various color components. For example, one CU may contain one luminance CB and two chroma CBs in the case of P and B slices of 4:2:0 chroma, and sometimes contain a single component CB (for example, one CU contains only one luminance CB or only two chrominance CBs in the case of I-images or I-slices). In some embodiments, in I-pictures or I-slices, the width or height of the TU is limited to a predetermined limit (eg, 64 for luma and 32 for chroma). If the width or height of the CB is greater than the limit value, then the TU is further divided until the size of the TU is less than or equal to the limit value.

[0126] In accordance with some embodiments, the QTBT partitioning scheme includes the following parameters:

CTU size: size of the root node of the quadtree;

MinQTSize: minimum size allowed for a leaf node of a quadtree;

MaxBTSize: the maximum allowable size of the root node of the binary tree;

MaxBTDepth: maximum allowed binary tree depth;

MinBTSize: The minimum allowable size of a binary tree leaf node.

[0127] In one example of the QTBT partitioning structure, the CTU size is set to 128x128 luminance samples with two corresponding blocks of 64x64 chrominance samples, MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize (for both width and height) is set to 4x4, and the value of MaxBTDepth is set to 4. The QTBT splitting structure is applicable to CTUs primarily for generating leaf quadtree nodes. The size of the leaf nodes of the quadtree can range from 16x16 (ie, MinQTSize) to 128x128 (ie, CTU size). If the size of the leaf node of the quadtree is 128x128, then the leaf node of the quadtree is not further split by the binary tree because the size exceeds MaxBTSize (ie, 64x64). On the other hand, the leaf node of the quadtree can be further decomposed by a binary tree. Thus, the leaf node of the quadtree is also the root node for the binary tree, and the leaf node of the quadtree has zero depth of the binary tree. If the depth of the binary tree reaches the MaxBTDepth value (for example, 4), then no further division is performed. If the width of a binary tree node is MinBTSize (eg 4), no further horizontal split is performed. Likewise, if the height of a binary tree node is MinBTSize, no further vertical splitting is performed. The leaf nodes of the binary tree are further processed by prediction and transformation without any further partitioning. In some embodiments, the maximum CTU size is 256x256 luminance samples.

[0128] The QTBT partition structure may also maintain separate QTBT structures for each of the luma and chrominance components. For example, for slices P and B, luminance and chroma CTBs in the same CTU may share the same QTBT structure. However, for I-slices, the luminance CTB is partitioned into CUs using the QTBT structure, and the chroma CTBs are partitioned into chroma CUs using a different QTBT structure. Thus, in this example, a CU in an I-slice contains a luma component coding block or two chroma component coding blocks, and a CU in a P or B slice contains coding units of all three chrominance components.

[0129] In some embodiments, inter-prediction for small blocks is limited to reduce memory access requirements for motion compensation, as a result, dual prediction is not supported for 4x8 and 8x4 blocks, and inter-prediction is not supported for 4x4 blocks. In other embodiments, the QTBT partitioning scheme does not include these restrictions.

[0130] In accordance with some embodiments, a multi-type tree (MTT) structure includes (i) quadtree splitting, (ii) binary tree splitting, and (iii) horizontal and vertical center-side ternary trees. In FIG. 9A shows an embodiment of a vertical center-side ternary tree, and FIG. 9B shows an example of a horizontal center-side ternary tree. Compared to the QTBT structure, the MTT structure can be a more flexible tree since additional structures are allowed.

[0131] Ternary tree splitting has significant advantages, such as providing a complement to quadtree splitting and binary tree splitting, where ternary tree splitting can capture objects located in the center of a block in which the quadtree and binary tree split at the center of the block. Another advantage of splitting a ternary tree is that the width and height of the sections of the proposed ternary trees are a power of 2, so no additional transformations are required. A two-level tree advantageously reduces the degree

difficulties. For example, the tree traversal complexity is T ^D , where T is the number of split types and D is the depth of the tree.

[0132] III. YUV formats

[0133] There are various YUV formats or chrominance formats (chroma sample formats), which are shown in FIG. 10A-10D. Each chrominance format may define different sampling or downsampling grids for different chrominance components.

[0134] IV. Color formats

[0135] The color of the video samples may be represented in various color formats or color spaces (eg, YCbCr or RGB). In the RGB format, the three components (ie, R, G, and B) are strongly correlated, resulting in statistical redundancy between the three color components. The color representation of video samples can be converted to another color space using linear transformations. Such an operation or process may be referred to as color space conversion.

[0136] Converting an RGB color space to a YUV color space can be done as follows:

Equation (1): Y=((66 * R+129 * G+25 * B+128)>>8)+16

Equation (2): U=((-38 * R - 74 * G+112 * B+128)>>8)+128

Equation (3): V=((112 * R - 94 * G - 18 * B+128)>>8)+128

[0137] Alternatively, converting an RGB color space to a YUV color space can be done as follows:

Equation (4): Y=round (0.256788 * R + 0.504129 * G+0.097906 * B)+16

Equation (5): U=round (-0.148223 * R - 0.290993 * G+0.439216 * B)+128

Equation (6): V=round (0.439216 * R - 0.367788 * G - 0.071427 * V)+128

[0138] V. Adaptive color conversion

[0139] The embodiments of the present invention may be used alone or in combination with each other in any order. In addition, each of the methods, encoders and decoders according to the disclosure of the present invention may be implemented by a processing circuit (eg, one or more processors or one or more integrated circuits). In one example, one or more processors execute a program recorded on a computer-readable medium. In accordance with embodiments of the disclosure of the present invention, the term block may be interpreted as a prediction block, a coding block, or a coding unit (ie, CU).

[0140] In accordance with embodiments of the disclosure of the present invention, the term luma component can refer to any color component encoded as the first component in coding order (eg, red (Red) or green (Green) chrominance component). Furthermore, in accordance with embodiments of the disclosure of the present invention, the term chrominance component can refer to any color component that is not encoded as the first component in coding order. For example, the second or third component may be referred to as the chrominance component.

[0141] In some embodiments, an annular color transform (also referred to as a color space transform) may be used to efficiently encode the content of an RGB video signal to control various characteristics of image blocks. Since the color transform can be used adaptively for different CUs, the color transform as a coding tool can be called Adaptive Color Transform (ACT). In some examples, the ACT may operate in the residual data area. A CU level flag may be transmitted to indicate the use of ACT.

[0142] For example, most screen content is captured in the RGB color space. For an image block in the RGB color space, a strong correlation between the various color components can usually be maintained so that the color space transformation is suitable for de-redundancy of the inter-color components. However, for screen content, there may be many blocks of images with different features and very rich colors, resulting in a decrease in the correlation between color components. For these blocks, it may be more efficient to encode directly in the RGB color space. Accordingly, the color space transformation can be adaptively applied to image blocks with different characteristics.

[0143] For example, a direct ACT color conversion (eg, a conversion performed by an encoder) may be performed as follows to convert an image block in RGB color space to YCoCg color space:

Equation (7):

[0144] The inverse color transform ACT (eg, the inverse transformation performed by the decoder) can be performed as follows: Equation (8):

[0145] In FIG. 11 shows an example of an encoder (1100) for performing color space conversion. In FIG. 11, the execution of the prediction precedes the application of the color space transform. For example, inter prediction or intra prediction is performed on the current block to generate a residual signal. The residual signal is supplied to the block (1102) direct color space conversion to perform direct conversion, for example, according to the formula (7). The output of the direct color space transform is fed to a cross component prediction block (1102) (CCP). The output of the block (1102) (SSR) is fed into the block (1106) transform (T, Transform) to perform a transformation such as a discrete cosine transform (DCT, Discrete Cosine Transform), in order to generate the coefficients of the transformation. The output of the transform block (1106) is fed to a quantizer (1108) (Q, Quantizer) to generate quantization coefficients. The quantization coefficients are provided to the entropy encoder block (1110) for converting the bits of the quantized coefficients into bits of the bit stream. The entropy encoder block (1110) may receive intra prediction mode or motion vector (MV) information associated with the current block and entropy encode this information into a bitstream.

[0146] The encoder (1100) may also contain components for restoring the residual signal. For example, the quantized coefficients can be provided to block (1112) inverse quantizer (IQ, Inverse Quantizer). The output data of the inverse quantizer (IQ) can be fed into block (1114) inverse transform (IT, Inverse Transform). The output of the inverse transform (IT) block (1114) may be fed to the inverse CCP block (1116). The output of the inverse CCP block (1116) may be fed to the color space inverse transform block (1118), where an inverse color transformation, such as formula (8), may be performed to generate a reconstructed residual signal.

[0147] FIG. 12 shows an example of a decoder (1200) for converting a bit stream into a residual signal. The bit stream shown in FIG. 12 may be a bitstream generated by the entropy encoder (1110), an example of which is shown in FIG. 11. The bit stream may be provided to an entropy decoder block (1202). The output of entropy decoder block (1202) may be provided to inverse quantizer (IQ) block (1204). The output of the inverse quantizer (IQ) block (1204) may be fed to the inverse transform (IT) block (1206). The output of the inverse transform (IT) block (1206) may be fed to the inverse CCP block (1208). The output of the inverse CCP block (1208) may be fed to the color space inverse transform block (1210), where an inverse color transformation, such as formula (8) conversion, can be performed to generate a residual signal. Intra prediction or inter prediction may be performed to generate a prediction block that is combined with the residual signal to reconstruct the current block. The blocks disclosed in FIG. 11 and 12 may be implemented as software executable by a processor or by a circuit such as an ASIC designed to perform the functions of each block.

[0148] In the examples in FIG. 11 and 12, the SSR operation may further be performed in the spatial area of the residual data. For example, the residual data of the second or third color component (also referred to as chrominance components) may be predicted based on the residual data of the first color component (also referred to as luma component) in the SSR. If ACT is not used, then CCP can significantly improve coding efficiency if the coding video data is in RGB color format. If ACT is used, then CCP may still be used in combination with ACT to improve the coding efficiency of video signals in the RGB color space.

[0149] In the examples in FIG. 11 and 12, the conversion operation may be further performed, for example, depending on the characteristics of the CU. For example, for a certain type of screen content, not using a transform operation may have a positive effect on coding efficiency compared to the case of using a transform. For example, a CU level conversion skip flag may be reported to indicate whether a conversion is skipped for the CU. Likewise, the quantization operation may also be optional. For example, if lossless operation is configured for some types of video content (medical images, wireless local screen sharing), then conversion and quantization may be omitted for certain CUs. For example, at the CU level, a skip transform and quantize CU flag may be reported to indicate the use of a lossless mode.

[0150] According to some embodiments, if the maximum transform size is not less than the width or length of one coding unit (CU), then one end node CU is also used as a transform processing element. For example, the ACT flag is signaled to one CU in order to select a color space for coding its residual signals. In addition, for CUs with Inter and Intra Block Copy (IBC, Inter and Intra Block Copy), ACT is allowed if there is at least one non-zero coefficient in the CU. For intra-coded CUs, ACT is enabled if the chrominance components select the same intra-prediction mode as the luma component, i.e., the DM mode. In an embodiment, QP adjustments, such as (-5, -5, -3), are applied to the conversion residuals to compensate for the change in dynamic range of the residuals before and after the color conversion.

[0151] Since the forward and reverse color conversions require access to the residues of all three components, according to some embodiments, ACT is prohibited in some scenarios in which not all of the residues of the three components are available. For example, if splitting a single tree is used, then the luma and chrominance samples within the same CTU are split into three different structures. As a result, the CUs in the luma tree contain only a luma component, and the CUs in the chrominance tree contain only two chrominance components. If Intra Sub-partition Prediction (ISP) is applied, then ISP is applied to luminance while chrominance signals are encoded without division. With the exception of the last ISP subdivisions, the other subdivisions contain only the luma component. Accordingly, ACT may be disabled for splitting a single tree (dual tree) and ISP.

[0152] VI. Linear cross-component model (CCLM, Cross-Component Linear Model)

[0153] For the chrominance component of the intra-coded PU, the encoder can select the best chrominance prediction modes from 8 modes, including flat, DC, horizontal, vertical, direct copy of the intra prediction mode (DM) based on the luma component, left and upper linear cross-component model (LT CCLM, Left and Top Cross-component Linear Mode), left linear cross-component model (L CCLM, Left Cross-component Linear Mode) and upper linear cross-component model (T CCLM, Top Cross-component Linear Mode). LT CCLM, L CCLM, and T CCLM can be divided into Cross-component Linear Mode (CCLM) groups. The difference between these 3 modes is that different areas of neighboring samples can be used to derive the parameters α and β. In the case of LT_CCLM, both left and upper adjacent samples can be used to derive the parameters α and β. In the case of L_CCLM, only left adjacent samples can be used to derive a parameters α and β. In the case of T_CCLM, only the upper adjacent samples can be used to derive the parameters α and βp.

[0154] Linear cross-component model (CCLM) prediction modes can be used to reduce cross-component redundancy, with chrominance samples predicted from reconstructed luminance samples of the same CU using a generic linear model as follows:

;Equation (9):

;

where pred _c (i,j) represents the predicted chrominance samples in the CU, and rec _L '(i,j) represents the downsampled reconstructed luma samples of the same CU. The parameters α and β can be obtained using a linear expression, which can also be called the minimax method. Since this computational process may be performed as part of the decoding process, not just as an encoder lookup operation, the syntax for passing the values of α and β may not be used.

[0155] For the 4:2:0 chrominance format, the CCLM prediction process may apply a six-tap interpolation filter to obtain a downsampled luma sample corresponding to the chrominance sample, as shown in FIG. 10B-10D. Using the formula, the downsampled luminance sample Rec'L[x, y] is calculated based on the reconstructed luminance samples. The downsampled luminance samples can be used to find the maximum and minimum sample points. The 2 points (luma and chrominance pair) (A, B) may represent the minimum and maximum values in a set of adjacent luma samples, as shown in FIG. 13.

[0156] FIG. 13 shows that the parameters α and β of the linear model can be obtained in accordance with the following expressions:

Equation (10):

Equation (11): β=γ _A - αx _A

[0157] The division operation in the example is avoided by using a multiplication and a shift operation. A single look-up table (LUT, Look-Up Table) can be used to store the precomputed values, and the absolute values of the difference between the maximum and minimum brightness samples can be used to indicate the index of the entry in the LUT. The LUT size can be, for example, 512.

[0158] FIG. 14A and 14B show an example of sample locations used to derive α and β values in LT_CCLM. In T_CCLM mode, in some examples, only the upper adjacent samples (including 2 * W samples) are used to compute the linear model coefficients. In FIG. 15A and 15B show an example of sample locations used to derive α and β values in T_CCLM.

[0159] In the L_CCLM mode, in some examples, only left adjacent samples (including 2*H samples) are used to compute the linear model coefficients. In FIG. 16A and 16B show an example of sample locations used to derive α and β values in L_CCLM.

[0160] The CCLM prediction mode may also include prediction between two chrominance components (ie, the Cr component is predicted based on the Cb component). Instead of using the reconstructed sample signal, Cb-Cr prediction in CCLM can be applied in the residual region. Cb-Cr prediction in CCLM can be implemented by adding the weighted reconstructed difference Cb to the original intra-prediction Cr to form the final Cr prediction:

Equation (12):

[0161] The luminance-chrominance prediction mode in CCLM may be added as one additional intra chrominance prediction mode. On the encoder side, one or more Rate Distortion (RD) cost checks are added for the chrominance components in order to select the intra chrominance prediction mode. If intra-prediction modes other than luma-chrominance in CCLM are used for CU chrominance components, then Cb-Cr prediction in CCLM is used to predict the Cr component.

[0162] Multiple Model CCLM (MMLM, Multiple Model CCLM) is another extension of CCLM in which there can be more than one model (eg, two or more models). In MMLM, adjacent luminance samples and adjacent chrominance samples of the current block may be divided into two groups, with each group being used as a training sequence to obtain a linear model (i.e., specific values of α and β are derived for a particular group). In addition, the samples of the current luminance block may also be classified based on the same rule for classifying neighboring luma samples.

Threshold (пороговое значение), классифицируется как группа 1; в то время как соседний отсчет, для которого Rec'L[x,y]>Threshold (пороговое значение), классифицируется как группа 2.[0163] FIG. 17 shows an example of dividing adjacent samples into two groups. The threshold value shown in FIG. 17 may be calculated as the average of adjacent reconstructed luminance samples. Neighbor sample for which Rec'L[x,y]

Threshold (threshold value), classified as group 1; while an adjacent sample for which Rec'L[x,y]>Threshold (threshold) is classified as group 2.

Equation (13):

[0164] VII. Examples of ACT applied to the prediction process

[0165] According to some embodiments, a color transform (such as ACT) may be applied before the encoder's prediction process and after the decoder's reconstruction process. At the encoder, ACT may be performed before prediction (eg, inter prediction, intra prediction), and reference samples as well as input source samples may be converted to a different color space if ACT is applied to the current CU. To restore the pixels at the decoder, if ACT is applied to a block being restored, the reference samples may be converted to an alternate color space before being used for prediction, and the recovered samples may then be back-transformed to the original color space.

[0166] In FIG. 18 shows an embodiment of the encoder and decoder processes when using ACT. The blocks shown in Fig. 18 may be implemented as software executable by a processor or by a circuit such as an ASIC designed to perform the functions of each block disclosed in FIG. 18. In the encoder, the ACT blocks (1800) and (1804) perform the ACT transformation of the reference signal and the input signal, respectively. The ACT performed in the encoder by ACT units (1800) and (1804) may be an ACT performed according to expression (7). The output of the ACT block (1800) is provided to the prediction (P) block (1802). In addition, a reference signal is applied to the prediction (P) block (1806). Blocks (1802) and (1806) prediction (P) may perform inter prediction or intra prediction. The transform (T) block 1808 receives one of the following signals: (i) the difference between the output of the predictor (P) block (1802) and the output of the ACT block (1804), and (ii) the difference between the output of the predictor (1806) and the input signal. A transform (T) block (1808) may perform a transform operation such as a discrete cosine transform (DCT). The output of the block (1808) transform (T) is fed to the quantizer (1810) (Q) to perform a quantization operation to form a set of coefficients.

[0167] At the decoder, an inverse quantizer (IQ) block (1812) receives coefficients to perform an inverse quantization process. The output of the inverse quantizer (IQ) block (1812) is fed to the inverse transform (IT) block (1814) to perform the inverse transformation. The ACT block (1820) receives the sum of the output of the prediction (P) block (1818) and the output of the inverse transform (IT) block (1814). Block (1816) ACT receives the reference signal and performs ACT to generate an input signal for the block (1818) prediction (P). Block (1816) ACT may perform color conversion, for example, in accordance with the expression (7). Block (1820) ACT may perform an inverse color conversion, for example, in accordance with the expression (8). Blocks (1818) and (1822) prediction (P) may perform inter prediction or intra prediction. The reconstructed original signal is provided as the output of the ACT block (1820) or a combination of the outputs of the prediction (P) block (1822) and the inverse transform (IT) block (1814).

[0168] VIII. Modified color conversion equations

[0169] In accordance with some embodiments, during the ACT process, the 2nd and 3rd color components (chrominance components) may also be shifted by a constant value "c" after and before color conversion for forward and reverse conversion, respectively. Equation (14) illustrates the modified forward transformation, and equation (15) illustrates the modified inverse (ie, inverse) transformation.

Equation (15):

Equation (16):

[0170] In accordance with some embodiments, the constant "c" is defined as l"(bitDepth-1), where bitDepth denotes the bit depth of the input sample.

[0171] IX. Interaction examples between ACT and other coding tools

[0172] In some embodiments, in order to enable in-loop color conversion to efficiently encode an input video signal using the RGB format, it is necessary to manage the interactions between color conversion (in the residual or pre-prediction) and multiple coding means. Examples of such coding tools are the linear cross-component model (CCLM) and dual tree partitioning. Embodiments of the present invention provide preferred color conversion control characteristics in view of certain coding means.

[0173] In some embodiments, a color transform is only applicable if the different color components are encoded using the same transform split tree. According to one embodiment, if the dual tree is applicable to the intra-slice, the color transform is only applicable to the inter-slice.

[0174] In some embodiments, if color transformation is applicable (eg, in the spatial residual domain), the CCLM mode is not applied or signaled because the generation of residual samples of one component depends on the reconstruction of another component. In another embodiment, if the CCLM mode is used, color conversion (eg, in the residual area) is not applied or signaled. In one embodiment, if color conversion is applicable on the intra-coded residual samples, the CCLM mode is not applied or signaled because the generation of the residual samples of one component depends on the reconstruction of another component. In one embodiment, if a color transform is applied on the residual samples and the CCLM mode is used, no color transform (eg, before prediction) is applied or signaled.

[0175] According to some embodiments, color conversion information is signaled for each CTU that is the largest coding unit (CU). In some embodiments, color conversion information is signaled and applied only to intra-coded blocks or only to externally coded blocks. In some embodiments, if a color transform is applicable, the dual tree is not applied (ie, different color components share the same transform binning).

[0176] According to some embodiments, for the current CU, if the bit depth of the luminance sample array, referred to as BitDepth Y, differs from the bit depth of the chroma sample array, referred to as BitDepth C, the color transformation (e.g., in the spatial domain of the residual or before prediction) is not used on the CU. For example, the current CU may be split based on an image with a 4:4:4 color format (RGB color format), however, the luma component and the chrominance component are represented at different bit depths. For example, the bit depth of the luminance component may be 10 bits, while the bit depth of the chrominance component may be 8 bits. The color conversion by applying different color conversion expressions (or color space conversion expressions) disclosed in this specification may not be effective in reducing the statistical relationship between luma and chrominance components. In such a scenario, the decoder may determine that no color conversion is used for the current CU if a syntax element(s) is received indicating different bit depths for different color components.

[0177] According to an embodiment, if all skip transform flags reported at the TU layer (or CU layer) for each color component index in the current CU are 1 or 0, the QP for each component of the current CU is 4, and the bit depth of the luminance components and chromaticity is different, then the color transform (in the spatial domain of the residual or before prediction) is not used for the CU. For example, skip transform flags for luminance or chrominance components indicate whether a transform (eg, DCT transform) is applied to the corresponding transform unit. A conversion skip flag of 1 indicates that the conversion is not applied to the corresponding conversion unit. A transform skip flag of 0 indicates that the decision to apply the transform to the corresponding transform unit depends on other syntax elements.

[0178] For example, if there is no conversion skip flag, then the following assumptions are made. If the block-based differential pulse code modulation (BDPCM) flag of the corresponding color component is 1, the skip transform flag is assumed to be 1. Otherwise (the BDPCM flag is 0), the skip transform flag is assumed to be 0. For example, the flag BDPCM equal to 1 indicates that BDPCM is applied to the current chrominance or luminance coding units, i.e. the transformation is omitted. The BDPCM flag of 0 indicates that BDPCM is not applied to the current chrominance or luminance coding units. For example, if the BDPCM flag is absent, then the BDPCM flag is assumed to be 0.

[0179] For example, if the QP for each CU component is 4, then the corresponding quantization step for the quantization transform coefficients may be set to 1. As a result, after the quantization operation, the values of the original transform coefficients are not changed. Accordingly, a QP value of 4 means the use of a lossless mode.

[0180] According to an embodiment, if all skip conversion flags reported at the TU level (or CU level) for each color component index for the current CU are 1 or 0, and the QP for each component of the current CU is 4, the color conversion (in spatial domain of the residual or before prediction) is not used for the current CU.

[0181] According to an embodiment, if all skip transform flags reported at the TU layer (or CU layer) for each color component index for the current CU are 1 or 0, no color transform (in the residual spatial domain or before prediction) is used for current CU.

[0182] According to some embodiments, two color space conversion options are supported for encoding or decoding processing. The encoder may choose one of two options to perform the color conversion, eg based on rate-distortion cost estimate, and report the choice in the bitstream. Alternatively, the encoder may inform the decoder (eg, by passing a flag) that a color transform is being applied to the CU. The decoder may determine which of the two options is used for the CU based on a set of conditions. Information about these conditions may be obtained or inferred based on one or more syntax elements transmitted in the bitstream.

[0183] In some examples, the inverse color space transform in two color transform options can be defined as follows:

Option 1

Equation (17): tmp=rY[x][y] (rCb[x][y]>>1)

Equation (18): rY[x][y]=tmp+rCb[x][y]

Equation (19): rCb[x][y]=tmp (rCr[x][y]>>1)

Equation (20): rСr[x][y]=rCb[x][y]+rСr[x][y]

Option 2

Equation (21): tmp=rY[x][y]-rCb[x][y]

Equation (22): rY[x][y]=rY[x][y]+rCb[x][y]

Equation (23): rCb[x][y]=tmp-rCr[x][y]

Equation (24): rCr[x] [y]=tmp+rCr[x][y],

where the input data for each color conversion option can be arrays of residual samples: an array of residual luminance samples of the residual luminance block, denoted as rY, an array of residual chrominance samples of the first residual chrominance block, denoted as rCb, and an array of chrominance residual samples of the second chrominance residual block, denoted like rCr. [x][y] indicate the coordinates of the residual samples in the corresponding residual block. For example, x can be an integer ranging from 0 to the box width value minus 1, and y can be an integer ranging from 0 to the box height value minus 1.

[0184] For example, two color conversion options, option 1 and option 2, may be used in the color space inverse transform block (1210) shown in FIG. 12 to convert the luminance and chrominance residual blocks to a modified version of the luminance and chrominance residual blocks.

[0185] According to the disclosure of the present invention, the color transform corresponding to the first option, that is, option 1, may be suitable for lossless CU processing, while the color conversion corresponding to the second option, that is, option 2, may be suitable for processing CU with losses. For example, for a lossless enabled CU (e.g., a skip transform and quantization flag of 1 is signaled or interpreted), the color transform according to option 1 may provide a better decorrelation effect compared to the color transform according to option 2.

[0186] According to an embodiment for the current CU, if a color conversion is applied and signaled at the CU level, all skip conversion flags reported at the CU level (or at the TU level) for each component index are 1, and the QP of each component of the current CU is 4, it can be determined that it is possible for the CU to apply the color transform according to option 1. For example, the color transform enable flag at the CU level can be received at the decoder from the bitstream. A color conversion enable flag of 1 indicates that the decoded residual data of the current coding unit is usable using a color space conversion. A color conversion enable flag of 0 indicates that the decoded residual data of the current coding unit has been processed without color space conversion. In an embodiment, if the color conversion enable flag is not present, the color conversion enable flag is assumed to be 0. One or more syntax elements indicating that the QP value is 4 may also be received in the bitstream. Based on the values of the CU-level color transform enable flag, the transform skip flag, and the QP value, the decoder can determine that the color transform according to option 1 is applied to the current CU.

[0187] According to an embodiment, for the current CU, if a color transform is applied and signaled at the CU layer, and at least one of the transform skip flags reported at the CU layer (or TU layer) for each component index is not equal to 1, it can be determined that the color transform according to option 2 is applied to the current CU. According to the embodiment, for the current CU, if the color transform is applied and signaled at the CU level, and the QP of one of the components of the current CU is not equal to 4, it can be determined, that for the current CU the color conversion according to option 2 is applied.

[0188] According to an embodiment, for the current CU, if a color transform is applied and signaled at the CU level, and all skip transform flags reported at the CU level for each component index are all 1, it can be determined that the transform is applied to the current CU. colors according to option 1.

[0189] According to an embodiment, only color conversion option 1 is applicable for CTU, slice, video sequence, and the like. Color conversion option 2 is not used. According to an embodiment, only color transform option 1 is applicable to a CTU, slice, video sequence, and the like, regardless of the values of the transform bypass and quantization flags. For example, a transform and quantization bypass flag of 1 indicates a transform and quantization bypass for the current CU. Thus, the current CU can be processed in a lossless mode. A transform and quantization bypass flag of 0 indicates that the transformation and quantization is not blocked for the current CU, or that the decision to bypass the transform and quantization for the current CU depends on other information.

[0190] According to an embodiment, the choice of color conversion option (option 1 or option 2) depends on the skip conversion flag(s) and/or BDPCM flag(s) of the current CU. For example, the choice of color conversion option may depend on the skip conversion flags of each color component of the current CU, as described above.

[0191] For example, the choice of color conversion option may depend on the BDPCM flag of the current CU. For example, if the BDPCM flag indicates applying the BDPCM mode to the luminance or chrominance components of the current CU, then it can be determined that transform processing is omitted for the corresponding luminance or chrominance component. On the other hand, if the BDPCM flag indicates that the BDPCM mode is not applied to the luminance or chrominance components of the current CU, then it can be determined that the transform is applied to the corresponding luminance or chrominance component. The skip conversion flags can be interpreted as zero or one, respectively. Accordingly, the selection of a color conversion option as described above may depend on the skip flags for converting each color component using interpreted values.

[0192] For example, the BDPCM mode is used to process the current CU. In the process of predicting the current pixel X, to the left of which is the neighboring pixel A, from the top is the neighboring pixel B, and to the lower left is the neighboring pixel C, the prediction P(X) can be determined as follows:

The prediction process uses unfiltered reference pixels in case of prediction of the top row and left column of the current CU. The prediction process then uses the reconstructed pixels for the remainder of the current CU. Pixels are processed in raster scan order within the CU. The prediction errors can then be quantized in the spatial domain without transformation followed by entropy coding. For example, a BDPCM flag may be transmitted at the CU level to indicate whether regular intra coding or BDPCM is being used.

[0193] X. Color Conversion Decoding Process Examples

[0194] FIG. 19 shows an embodiment of a process (1900) performed by a decoder, such as decoder (710). The process (1900) may start from a step (S1901) followed by a step (S1910).

[0195] In step (S1910), a syntax element may be received from the encoded video bitstream a. For example, the syntax element may be a CU-level color conversion enable flag indicating that the residual blocks of the current CU have been processed using a color space conversion. The residual blocks of the current CU may include a luma residual block, a first chrominance residual block, and a second chrominance residual block.

[0196] In step (S1920), in response to receiving a syntax element indicating that the residual blocks of the current CU are processed using color space conversion, one of two variants of color space conversion equations can be selected. For example, two versions of the color space conversion equations may correspond to the color space conversion option 1 and option 2 represented by expressions (17)-(20) and expressions (21)-(24). In other examples, the two color space conversions may take other forms corresponding to different color conversion expressions.

[0197] For example, one of the two variants of the color space conversion equations may be selected in accordance with the value of the skip conversion flag corresponding to each of the residual blocks of the current CU. Skip transform flags may be received from the encoded video bitstream. On the other hand, the skip transform flags may be interpreted based on other information if the skip transform flags are not reported in the encoded video bitstream.

[0198] For example, one of the two variants of the color space conversion equations may be selected in accordance with the QP value of each of the residual blocks of the current CU. For example, if the QP value is 4 and the conversion skip flags of each of the residual blocks are 1, then color conversion option 1 may be selected. If the QP value is not equal to 4, then color conversion option 2 may be selected.

[0199] For example, the choice of one of the two color conversion options may depend on the BDPCM flag of the current CU. For example, if the BDPCM flag indicates that a color component of the current CU is in BDPCM mode, the skip transform flag of that color component of the current CU may be interpreted as a flag with the value 1. The choice of one of the two options may be based on the value of the skip transform flag in combination with the flag values of the other color components of the current CU.

[0200] In step (S1930), an inverse color space transform using the selected version of the color space transform equations may be applied to the residual blocks of the current CU to generate modified versions of the residual blocks of the current CU. The process (1900) may end at step (S1999).

[0201] XI. computer system

[0202] The techniques described above may be implemented in computer software using computer-readable instructions that are physically stored on one or more computer-readable media. For example, in FIG. 20 shows a computer system (2000) suitable for implementing certain embodiments of the subject matter of the present invention.

[0203] Computer software may be coded using any suitable machine code or programming language that can be assembled, compiled, linked, or similar mechanisms to form code containing instructions that can be directly executed or interpreted, executed as microinstructions, etc. .p. one or more computer central processing units (CPU, Central Processing Unit), graphic processors (GPU, Graphics Processing Unit), etc.

[0204] Instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, IoT devices, and the like.

[0205] The components shown in FIG. 20 for a computer system (2000) are exemplary in nature and are not intended to limit the scope of use or functionality of computer software implementing the disclosure of the present invention in any way. In addition, the configuration of the components should not be considered in terms of any dependency or any requirement associated with any component or combination of components shown within the exemplary computer system implementation (2000).

[0206] The computer system (2000) may include user interface input devices. Such a user interface input device may be responsible for the input of information by one or more users via, for example, tactile input (such as keystrokes, screen shift, information glove movements), audio input (e.g., voice, claps), visual input (e.g., , gestures), olfactory input (not shown in the drawing). User interface devices may also be used to capture certain media information not necessarily directly related to the user's conscious input, such as audio (e.g. speech, music, audio), images (e.g. scanned images, photographs taken with a capture camera). still images), video information (such as 2D video, 3D video including stereoscopic video).

[0207] User interface devices may include (the drawing shows only one device of each type): keyboard (2001), mouse (2002), touch pad (2003), touch screen (2010), information glove (not shown in the drawing), joystick (2005), microphone (2006), scanner (2007), camera (2008).

[0208] The computer system (2000) may also include user interface output devices. Such user interface output devices may stimulate the experience of one or more users through, for example, tactile output, sound, light, and smell/taste. Such user interface output devices may include tactile output devices (e.g., haptic feedback via a touch screen (2010), an information glove (not shown), or a joystick (2005), but there may also be tactile feedback devices that are not input), audio output devices (such as speakers (2009), headphones (not shown in the drawing), visual output devices (such as screens (2010), including CRT screens, liquid crystal screens, plasma screens, OLED screens, each of which can be equipped with touch input, tactile feedback, some of which can output two-dimensional images or images with a dimension greater than three dimensions, through such means as stereographic output; virtual reality glasses (not shown in the drawing), holographic displays and smoke tanks (not shown)), and printers (not yet zany on the drawing).

[0209] The computer system (2000) may also include user-accessible storage devices and associated media such as optical media including CD/DVD ROM/RW (2020) with CD/DVD or similar media (2021), a flash drive (2022), removable or non-removable hard drive (2023), traditional magnetic media such as a floppy disk (not shown in the drawing), specialized ROM/ASIC/PLD based devices such as dongles (not shown in the drawing), etc. .P.

[0210] Those skilled in the art will appreciate that the term "computer-readable medium" as used in connection with the subject matter of the present invention does not encompass transmission media, carrier signals, or other transient signals.

[0211] The computer system (2000) may also include an interface (2054) with one or more communication networks (2055). Networks, for example, can be wireless, wired, optical. Networks can also be local, global, urban, transport and industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., wired or wireless digital wide area networks including cable TV, satellite TV and terrestrial television broadcasting. , transport and industrial networks, including CANBus, etc. Certain networks typically require external network interface adapters that connect to specific general purpose data ports or peripheral buses (2049) (such as computer system USB ports (2000)); others are typically integrated into the core of the computer system (2000) by connecting to the system bus as described below (eg, an Ethernet interface on a personal computer or a cellular network interface on a smartphone computer system). When using any of these networks, the computer system (2000) can communicate with other entities. Such communication can be unidirectional, in the receive direction only (for example, television broadcasts), unidirectional, in the transmit direction only (for example, CANbus with certain CANbus devices), or bidirectional, for example, with other computer systems using local or wide area networks. Certain protocols and protocol stacks may be used on each of these networks and network interfaces, as described above.

[0212] The above user interface devices, user accessible storage devices and network interfaces can be connected to the core (2040) of the computer system (2000).

[0213] The core (2040) may contain one or more central processing units (CPU, Central Processing Unit) (2041), graphics processors (GPU, Graphics Processing Unit) (2042), specialized programmable processing units in the form of field programmable gate arrays (FPGA , Field Programmable Gate Area) (2043), hardware accelerators (2044) for certain tasks, graphics adapters (2050), etc. These devices, together with read-only memory (ROM) (2045), random access memory (2046), internal mass storage devices such as internal non-user-accessible hard drives, SSDs (2047), etc., can be connected via the system bus (2048) . In some computer systems, the system bus (2048) may be accessed through one or more physical connectors, allowing the system to be expanded by adding a CPU, GPU, or the like. Peripherals can be connected either directly to the system bus (2048) of the kernel, or via a peripheral bus (2049). For example, a screen (2010) can be connected to a graphics adapter (2050). Peripheral bus architectures include PCI, USB, and the like.

[0214] CPUs (2041), GPUs (2042), FPGAs (2043), and accelerators (2044) may execute certain instructions, which together may form the computer code mentioned above. This computer code may be stored in ROM (2045) or RAM (2046). Intermediate data may also be stored in RAM (2046), while persistent data may be stored, for example, in an internal mass storage device (2047). Fast writing and retrieval of data from any storage device can be performed using cache memory, which can be closely associated with one or more CPU (2041), GPU (2042), internal mass storage devices (2047), ROM (2045), RAM (2046), etc.

[0215] The computer-readable medium may contain computer code for performing various operations implemented on the computer. The media and computer code may be specially designed for the purposes of the present invention, or they may be well known and available to those skilled in the art of computer software.

[0216] For example, without being limited to the above example, a computer system (2000) of a certain architecture, and in particular, a kernel (2040), may provide system functionality as a result of execution by the processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) n.) software implemented on one or more computer-readable media. Such computer-readable media can be media associated with the user-accessible mass storage device mentioned above, as well as certain kernel media (2040), which are inherently durable storage media, such as mass storage (2047), or ROM (2045) . Software implementing various embodiments of the disclosure of the present invention may be stored on such devices and executed by the kernel (2040). The computer-readable medium may comprise one or more storage devices or microchips, as required. The software may initialize the core (2040) and in particular the processors located therein (including CPUs, GPUs, FPGAs, etc.) to execute specific processes or parts of specific processes described in this description, including the definition of data structures to RAM (2046) and modification of such data structures in accordance with the processes defined by the software. In addition, or alternatively, a computer system may provide functionality as a result of logical hard-coded or otherwise circuitry (e.g., accelerator (2044)) solutions that may operate instead of or in conjunction with software to perform specific processes or specific parts of specific processes given in this description. Reference to software may cover logic diagrams and vice versa where appropriate. Reference to a computer-readable medium may include a circuit (eg, an integrated circuit (IC)) that stores executable software, a circuit that implements executable logic instructions, or both, where appropriate. The disclosure of the present invention covers any suitable combination of hardware and software.

Appendix A: Abbreviations

ASIC: Application-Specific Integrated Circuit

BMS: Benchmark Set

CANBus: Controller Area Network Bus

CBF: Coded Block Flag

CCLM: Cross-Component Linear Model CD:

compact disc

CPUs: Central Processing Units, CRT Central Processing Units:

Cathode Ray Tube, cathode ray tube CTBs:

Coding Tree Blocks

CTUs: Coding Tree Units

CU: Coding Unit

DT: Dual Tree

DVD: Digital Video Disc

FPGA: Field Programmable Gate Areas, Field Programmable Gate Array

GOPs: Groups of Pictures

GPUs: Central Processing Units

GSM: Global System for Mobile Communications

HDR: High Dynamic Range, extended dynamic range

HEVC: High Efficiency Video Coding

HRD: Hypothetical Reference Decoder

1C: Integrated Circuit, integrated circuit

ISP: Intra Sub-Partitions

JVET: Joint Video Exploration Team

LAN: Local Area Network, LCD Local Area Network:

Liquid-Crystal Display, LTE Liquid Crystal Display:

Long Term Evolution, MPM Long Term Evolution Technology:

Most Probable Mode MV: Motion Vector

JEM: Joint Exploration Model

Organic Light-Emitting Diode, Organic LED PBs:

Prediction Blocks, units of prediction

PCI: Peripheral Component Interconnect

PLD: Programmable Logic Device

PU: Prediction Unit

QP: Quantization Parameter

RAM: Random Access Memory

ROM: Read-Only Memory

SBT: Sub-block Transform

SDR: Standard Dynamic Range

SEI: Supplementary Enhancement Information

SNR: Signal Noise Ratio

Solid-State Drive, TU Solid State Drive:

Transform Unit

USB: Universal Serial Bus, Universal Serial Bus VUI:

Video Usability Information, easy-to-use visual information

VVC: Versatile Video Coding, Advanced Video Coding WAIP:

Wide-Angle Intra Prediction

[0217] While the invention has been described in terms of several non-limiting embodiments, there are variations, permutations, and various replacement equivalents that do not fall outside the scope of the disclosure of the present invention.

Claims (50)

1. A video decoding method, including: receiving a syntax element from the encoded video data bitstream indicating whether residual blocks of the current coding unit (CU) have been processed using color space transformation, wherein the residual blocks of the current CU include a luma residual block, a first chrominance residual block, and a second chrominance residual block; in response to the syntax element indicating that the residual blocks of the current CU are processed using a color space transform, applying, based on the fact that the transform skip flag corresponding to each of the residual blocks of the current CU is 1, and the quantization parameter ( QP) corresponding to each of the residual blocks of the current CU is equal to 4, inversely transform the color space, using the first version of the color space transformation equations, to the residual blocks of the current CU to generate modified versions of the residual blocks of the current CU, and the first version of the color space transformation equations contains the following equations: tmp=rY[x][y]-(rCb[x][y]>>1), rY[x][y]=tmp+rCb[x][y], rCb[x][y]=tmp-(rCr[x][y]>>1), and rCr[x][y]=rCb[x][y]+rCr[x][y], while the input data for the first version of the color space conversion equations include: array of residual brightness readings of the residual brightness block containing elements rY[x][y], an array of chrominance residual samples of the first chrominance residual block containing the elements rCb[x][y], and an array of residual chrominance samples of the second residual chrominance block containing the elements rCr[x][y]; and the output for the first variant of the color space conversion equations includes: modified array of residual brightness samples of the residual brightness block containing elements rY[x][y], a modified array of residual chrominance samples of the first residual chrominance block containing the elements rCb[x][y], and modified array of residual chrominance samples of the second residual chrominance block containing elements rCr[x][y]. 2. The method according to p. 1, also including: in response to the syntax element indicating that the residual blocks of the current CU have been processed using a color space transform, selecting one of two variants of the color space transform equations, and two variants of the color space conversion equations include: the first version of the color space transformation equations and the second version of the color space transformation equations containing the following equations: tmp=rY[x][y]-rCb[x][y], rY[x][y]=rY[x][y]+rCb[x][y], rCb[x][y]=tmp-rCr[x][y], and rCr[x][y]=tmp+rCr[x][y], while the input data for the second version of the color space conversion equations include: array of residual brightness readings of the residual brightness block containing elements rY[x][y], an array of chrominance residual samples of the first chrominance residual block containing the elements rCb[x][y], and an array of residual chrominance samples of the second residual chrominance block containing the elements rCr[x][y]; and the output for the second variant of the color space conversion equations includes: modified array of residual brightness samples of the residual brightness block containing elements rY[x][y], a modified array of residual chrominance samples of the first residual chrominance block containing the elements rCb[x][y], and modified array of residual chrominance samples of the second residual chrominance block containing elements rCr[x][y]. 3. The method according to claim 2, characterized in that the choice of one of the two variants of the color space conversion equations includes selecting a second variant of the color space conversion equations if at least one of the skip conversion flags corresponding to the residual blocks of the current CU is 0. 4. The method according to claim 2, characterized in that the choice of one of the two variants of the color space conversion equations includes selecting the second variant of the color space transformation equations if at least one of the QPs corresponding to the residual blocks of the current CU is not equal to 4. 5. The method according to p. 2, including also: receiving a syntax element indicating that only the first version of the color space transformation equations is applied, wherein the syntax element is one of the following elements: a slice level syntax element, an image level syntax element, or a sequence level syntax element. 6. The method according to p. 2, including also receiving a syntax element that indicates that only the first version of the color space transformation equations is applied, regardless of the value of the flag indicating whether transformation and quantization for the residual CUs are skipped, where the syntax element is one of the following elements: level syntax element slice, image-level syntax element, or sequence-level syntax element. 7. The method according to claim 1, characterized in that the choice of one of the two variants of the color space conversion equations includes selecting one of the two variants of the color space conversion equations according to a value indicating whether block-based differential pulse code modulation (BDPCM) is applied to the current CU. 8. The method according to p. 1, including also determining that a color space transform is not applied to the residual blocks of the current CU if the current CU's luma sample bit depth is different from the current CU's chrominance sample bit depth. 9. The method according to p. 1, including also determining that a color space transform is not applied to the residual CUs if the luminance bit depth of the CU is different from the chrominance sample bit depth of the CU and the transform skip flag corresponding to each of the residual CUs is 1 or 0. 10. The method according to p. 1, including also determining that a color space transform is not applied to the residual CUs if the luminance bit depth of the CU is different from the chroma sample bit depth of the CU, the QP corresponding to each of the residual CUs is 4, and a transform skip flag corresponding to each of the residual CUs , is equal to 1 or 0. 11. A device for video decoding, containing a circuit configured to implement the method according to any one of paragraphs. 1-10. 12. A computer-readable storage medium containing instructions, during the execution of which the processor performs the method according to any one of paragraphs. 1-10.

Images