RU2786086C1 - Method and device for cross-component linear modeling for internal prediction - Google Patents

Abstract

FIELD: video encoding.

SUBSTANCE: invention relates to means for video encoding. Restored brightness samples are obtained. Maximum and minimum values of brightness samples are determined based on restored brightness samples. The difference between maximum and minimum values of brightness samples is obtained. A position of the most significant bit of the difference between maximum and minimum values of brightness samples is determined. A value is extracted from a lookup table (LUT), using a set of bits as an index, wherein the set of bits follows the position of the most significant bit of the difference of maximum and minimum values of brightness samples. Parameters α and β of a linear model are obtained based on the extracted value. A predicted value of the brightness sample is calculated using obtained parameters α and β of the linear model.

EFFECT: increase in the efficiency of video encoding.

24 cl, 15 dwg, 4 tbl

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. provisional application no. 62/786,563, filed December 31, 2018, entitled "METHOD AND APPARATUS OF CROSS-COMPONENT LINEAR MODELING FOR INTRA PREDICTION", which is hereby incorporated by reference.

Technical field

Embodiments of the present application (disclosure) generally relate to the field of image processing and, in particular, to intra prediction using cross-component linear modeling.

State of the art

Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as digital television broadcast, video transmission over the Internet and mobile networks, real-time conversational applications such as video chat, video conferencing, DVD and Blu-ray discs, video content collection and editing systems; and video camera security applications.

The amount of video data required to display even a relatively short video can be substantial, which can lead to difficulties when the data must be streamed or otherwise transferred over a bandwidth limited communications network. Thus, video data is typically compressed prior to transmission over modern telecommunications networks. Video size can also be an issue when the video is stored on a storage device, as memory resources can be limited. Video compression devices often use source software and/or hardware to encode video data prior to transmission or storage, thereby reducing the amount of data needed to represent digital video images. The compressed data is then received at the destination by a video decompressor, which decodes the video data. With limited network resources and ever-increasing demands for higher video quality, improved compression and decompression techniques are desired that improve compression rates with little or no loss in image quality.

The essence of the invention

Embodiments of the present application provide devices and methods for encoding and decoding according to the independent claims.

The above and other objects are achieved by the essence of the independent claims. Additional forms of implementation are apparent from the dependent claims, description and figures.

According to a first aspect, the invention relates to a method for intra-prediction of a block chrominance sample by applying a cross-component linear model. The method includes: obtaining reconstructed brightness samples; determining maximum and minimum luminance sample values based on the reconstructed luminance samples; obtaining the difference between the maximum and minimum values of the brightness samples; determining the position of the most significant bit of the difference between the maximum and minimum values of the brightness sample. The method also includes: extracting a value from a look-up table (LUT) using a set of bits as an index, a set of bits following the position of the most significant bit of the difference between the maximum and minimum luminance sample values; obtaining parameters α and β of the linear model based on the extracted value; and calculating a predicted chrominance sample value using the obtained linear model parameters α and β.

According to the first aspect of the invention, the index for the LUT is calculated in an elegant way that extracts a few bits in the binary representation. As a result, the efficiency of extracting the value from the LUT is improved.

In an exemplary embodiment of the method according to the first aspect, as such, the method obtains the linear model parameters α and β by multiplying the extracted value by the difference between the maximum and minimum values of the reconstructed chrominance samples.

As the efficiency of extracting a value from the LUT increases, the value of the multiplier for obtaining the parameters α and β of the linear model is reduced to a minimum.

In an exemplary form of implementing the method according to the first aspect as such, the LUT includes at least two neighboring values stored in the LUT that correspond to different steps of the obtained difference, and the value of this step increases with the value of the difference or is a constant. The index for the LUT is calculated in an elegant way that extracts a few bits in the binary representation, and accordingly the size of the entry in the LUT corresponding to the index is minimized. As a result, the LUT size is minimized.

An apparatus for intra block chroma sample prediction by applying a cross-component linear model is provided according to a second aspect of the invention. The apparatus according to the second aspect of the invention includes an acquisition unit, a determination unit, and a calculation unit. An acquisition block configured to receive the reconstructed luminance samples. A determiner configured to determine maximum and minimum luminance sample values based on the reconstructed luminance samples. An acquiring block further configured to obtain the difference between the maximum and minimum luminance sample values. A determiner further configured to determine the position of the most significant bit of the difference between the maximum and minimum luminance sample values. A calculation unit configured to extract a value from a look-up table (LUT) using a set of bits as an index, a set of bits following the position of the most significant bit of the difference between the maximum and minimum luminance sample values, obtaining linear model parameters α and β based on the extracted value; and calculating a predicted chrominance sample value using the obtained linear model parameters α and β.

According to the second aspect of the invention, the device calculates an index for the LUT in an elegant way that extracts a few bits in the binary representation. As a result, the efficiency of extracting the value from the LUT is improved.

According to a third aspect, the invention relates to a device for decoding a video stream, including a processor and a memory. The memory stores instructions that cause the processor to execute the method according to the first aspect or any possible embodiment of the first aspect.

According to a fourth aspect, the invention relates to a device for encoding a video stream, including a processor and a memory. The memory stores instructions that cause the processor to execute the method according to the first aspect or any possible embodiment of the first aspect.

According to a fifth aspect, a computer-readable storage medium is provided that stores instructions that, when executed, call one or more processors configured to encode video data. The instructions cause one or more processors to execute the method according to the first aspect or any possible embodiment of the first aspect.

According to a sixth aspect, the invention relates to a computer program containing program code for executing the method according to the first aspect, or any possible embodiment of the first aspect, when executed on a computer.

Details of one or more embodiments are set forth in the accompanying drawings and in the description below. Other features, objects and advantages will be apparent from the description, drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Further embodiments of the invention are described in more detail with reference to the accompanying figures and drawings, in which:

Fig. 1A is a block diagram showing an example of a video coding system configured to implement embodiments of the invention;

1B is a block diagram showing another example of a video coding system capable of implementing embodiments of the invention;

Fig. 2 is a block diagram showing an example of a video encoder configured to implement embodiments of the invention;

Fig. 3 is a block diagram showing an exemplary structure of a video decoder configured to implement embodiments of the invention;

Fig. 4 is a block diagram illustrating an example of an encoding device or a decoding device;

Fig. 5 is a block diagram illustrating another example of an encoding device or a decoding device;

Fig. 6 is a drawing illustrating the concept of a cross-component linear model for intra chroma prediction;

Fig. 7 is a drawing illustrating a simplified method in linear model parameters;

Fig. 8 is a drawing illustrating a distribution of the probability of occurrence of a difference between the maximum and minimum values of the luminance reference samples;

Fig. 9 is a flowchart illustrating an exemplary lookup table generation process in accordance with an embodiment of the invention;

Fig. 10 is a block diagram drawing illustrating an embodiment of index derivation for an exemplary lookup table;

Fig. 11 is a block diagram illustrating exemplary block chroma sample intra prediction by applying a cross-component linear model;

Fig. 12 is a block diagram showing an exemplary structure of an apparatus for intra block chromaticity sample prediction by applying a cross-component linear model;

Fig. 13 is a block diagram showing an exemplary structure of a content delivery system 3100 that implements a content delivery service; as well as

Fig. 14 is a block diagram showing the structure of an example of a terminal device.

In the following, like reference numerals refer to like or at least functionally equivalent features, unless expressly stated otherwise.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following description, reference is made to the accompanying figures, which form part of the disclosure and show, by way of illustration, specific aspects of embodiments of the invention or specific aspects in which embodiments of the present invention may be used. It is clear that embodiments of the invention can be used in other aspects and contain structural or logical changes not shown in the figures. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

For example, it is understood that the disclosure in connection with the described method may also be true for the corresponding device or system configured to perform the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more blocks, such as functional blocks, for performing the one or more method steps described (for example, one block performing one or more steps, or a plurality of blocks, each of which performs one or more of the plurality of steps), even if such one or more blocks are not explicitly described or illustrated in the figures. On the other hand, for example, if a particular device is described in terms of one or more blocks, such as functional blocks, the corresponding method may include one step to perform the functionality of one or more blocks (for example, one step to perform the functionality of one or more blocks or a plurality of steps, each performing the functionality of one or more of the plurality of blocks), even if such one or more steps is not explicitly described or illustrated in the figures. In addition, it is understood that the features of the various exemplary embodiments and/or aspects described herein may be combined with each other, unless specifically indicated otherwise.

Video coding generally refers to the processing of a sequence of images that form a video or video sequence. Instead of the term "picture", the term "frame" or "picture" can be used as synonyms in the field of video coding. Video encoding (or encoding in general) consists of two parts: video encoding and video decoding. Video encoding is performed at the source side, typically comprising processing (eg, by compressing) the source video images to reduce the amount of data required to represent the video images (for more efficient storage and/or transmission). Video decoding is performed on the destination side (target side) and usually contains inverse processing compared to the encoder for recovering video images. Embodiments relating to "coding" video images (or images in general) should be understood as referring to "coding" or "decoding" video images or corresponding video sequences. The combination of an encoding part and a decoding part is also called CODEC (encoding and decoding).

In the case of lossless video coding, the original video images can be reconstructed, i.e. the reconstructed video images have the same quality as the original video images (assuming there is no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, additional compression is performed, for example by quantization, in order to reduce the amount of data representing video images that cannot be fully reconstructed in the decoder, i.e. the quality of the restored video images is lower or worse than the quality of the original video images.

Several video coding standards belong to the group of "hybrid lossy video codecs" (i.e., combine spatial and temporal sample-domain prediction and 2D transform coding to apply transform-domain quantization). Each picture of a video sequence is usually divided into a set of non-overlapping blocks, and encoding is usually done at the block level. In other words, in the encoder, video is usually processed, i.e. encoded at the block (video block) level, for example, using spatial (inside the image) prediction and/or temporal (external to the image) prediction to generate a prediction block by subtracting the prediction block from the current block. (of the block currently being processed/to be processed) to obtain the residual block, transform the residual block, and quantize the residual block in the transform domain to reduce the amount of data to be transmitted (compression), while in the decoder, the inverse processing is compared with encoder is applied to an encoded or compressed block to reconstruct the current block for presentation. In addition, the encoder duplicates the processing cycle of the decoder, so that both will generate identical predictions (eg, intra and inter predictions) and/or reconstructions for processing, i.e. encoding subsequent blocks.

In the following embodiments, the video coding system 10, video encoder 20, and video decoder 30 are described based on FIG. 1 to 3.

Fig. 1A is a schematic block diagram illustrating an exemplary coding system 10, such as a video coding system 10 (or short coding system 10), that may use the techniques of this present application. Video encoder 20 (or short encoder 20) and video decoder 30 (or short decoder 30) of video coding system 10 represent examples of devices that can be configured to perform techniques in accordance with the various examples described herein.

As shown in FIG. 1A, encoding system 10 includes a source device 12 configured to provide encoded image data 21 to, for example, target device 14 to decode encoded image data 21.

Source device 12 contains encoder 20 and may additionally, i. e. optionally, contain an image source 16, a preprocessor (or preprocessor) 18, such as an image preprocessor 18, and a communication interface or communication unit 22.

The image source 16 may comprise or be any type of image capturing device, such as a camera for capturing a real image, and/or any type of image generating device, such as a computer graphics processor for creating a computer animation image, or any other device for capturing and/or providing a real image. any type, computer generated image (e.g. screen content, virtual reality (VR) image) and/or any combination thereof (e.g. augmented reality (AR) image). The snapshot source can be any type of memory or storage that stores any of the above images.

Unlike the preprocessor 18 and the processing performed by the preprocessor 18, the image or image data 17 may also be referred to as raw image or raw image data 17.

The preprocessor 18 is configured to receive the (raw) image data 17 and perform pre-processing of the image data 17 to obtain a pre-processed image 19 or pre-processed image data 19. The pre-processing performed by the preprocessor 18 may, for example, include cropping, color format conversion (eg from RGB to YCbCr), color correction or noise reduction. It can be understood that the pre-processing unit 18 may be an optional component.

Video encoder 20 is configured to receive pre-processed image data 19 and provide encoded image data 21 (more details will be described below, for example, based on FIG. 2). Communication interface 22 of source device 12 may be configured to receive encoded image data 21 and transmit encoded image data 21 (or any additional processed version thereof) over communication channel 13 to another device, such as target device 14 or any other device, for storage. or direct recovery (recovery).

Target device 14 includes a decoder 30 (eg, video decoder 30) and may additionally, i. e. optionally, contain a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32) and a display device 34 .

The communication interface 28 of the target device 14 is configured to receive the encoded image data 21 (or any additional processed version thereof), for example, directly from the source device 12 or from any other source, for example, a storage device, for example, an encoded image data storage device, and provide encoded image data 21 to decoder 30.

Communication interface 22 and communication interface 28 may be configured to transmit or receive encoded image data 21 or encoded data 13 over a direct link between source device 12 and target device 14, such as a direct wired or wireless connection, or over any type of network, such as wired or wireless network, or any combination thereof, or any private and public network, or any combination thereof.

The communication interface 22 may, for example, be configured to package the encoded image data 21 into an appropriate format, such as packets, and/or process the encoded image data using any type of transmission coding or processing for transmission over a communication line or communication network.

Communication interface 28 analogous to communication interface 22 may, for example, be configured to receive transmitted data and process transmission data using any kind of appropriate decoding or transmission processing and/or decompression to obtain encoded image data 21.

Both communication interface 22 and communication interface 28 can be configured as unidirectional communication interfaces, as indicated by the arrow for communication channel 13 in FIG. 1A pointing from source device 12 to target device 14, or as bidirectional communication interfaces, and can be configured, for example, to send and receive messages, for example, to establish a connection, to acknowledge and exchange any other information related to the communication line and /or data transmission, for example, to the transmission of encoded image data.

The decoder 30 is configured to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (more details will be described below, for example, based on FIG. 3 or FIG. 5.

The post-processor 32 of the target device 14 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image 31, to obtain post-processed image data 33, such as the post-processed image 33. The post-processing performed by the post-processing unit 32 may include, for example , color format conversion (eg, from YCbCr to RGB), color correction, cropping or resampling, or any other processing, for example, to prepare the decoded image data 31 for display, for example, by the display device 34.

The display device 34 of the target device 14 is configured to receive post-processed image data 33 for displaying the image to, for example, a user or viewer. The display device 34 may be or comprise any kind of display for presenting the reconstructed image, such as an integrated or external display or monitor. The displays may, for example, comprise liquid crystal displays (LCD), organic light emitting diode (OLED) displays, plasma displays, projectors, micro LED displays, liquid crystals on silicon (LCoS), digital light processor (DLP), or any other displays.

Although FIG. 1A depicts source device 12 and target device 14 as separate devices, device embodiments may also contain both or both functionality, source device 12 or corresponding functionality and target device 14 or corresponding functionality. In such embodiments, source device 12 or related functionality and target device 14 or corresponding functionality may be implemented using the same hardware and/or software, or using separate hardware and/or software, or any of them. combinations.

As will be apparent to those skilled in the art based on the description, the presence and (exact) separation of the functionality of the various blocks or functionality in source device 12 and/or target device 14 as shown in FIG. 1A may vary depending on the actual device and application.

Encoder 20 (for example, video encoder 20) or decoder 30 (e.g. video decoder 30) or and encoder 20 and decoder 30 may be implemented through a processing circuit as shown in FIG. 1B, for example, one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding, or any combinations thereof. Encoder 20 may be implemented via processing circuitry 46 to implement various modules, as discussed with respect to encoder 20 in FIG. 2 and/or any other system or subsystem described herein. Decoder 30 may be implemented via processing circuitry 46 to implement various modules, as discussed with respect to decoder 30 in FIG. 3 and/or any other decoder system or subsystem described herein. The processing circuit may be configured to perform various operations, as described below. As shown in FIG. 5, if the techniques are partially implemented in software, the device may store instructions for the software on a suitable non-volatile computer-readable storage medium and may execute the instructions in hardware using one or more processors to execute the techniques of this disclosure. Either of video encoder 20 and video decoder 30 may be integrated as part of a combined encoder/decoder (CODEC) in one device, for example, as shown in FIG. 1B.

Source device 12 and destination device 14 may comprise any of a wide range of devices, including any type of portable or stationary device, such as laptops or laptops, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, game consoles, video streaming devices (eg, content service servers or content delivery servers), broadcast receiver device, broadcast transmitter device, and the like. and can use any or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and target device 14 may be wireless communication devices.

In some cases, the coding system 10 the video illustrated in Fig. 1A is merely an example, and the methods of the present application may apply to video coding settings (eg, video coding or video decoding) that do not necessarily involve data transfer between encoding and decoding devices. In other examples, data is retrieved from local memory, transferred over a network, and the like. The video encoding device may encode and store data in memory and/or the video decoder may retrieve and decode data from memory. In some examples, encoding and decoding is performed by devices that do not communicate with each other, but simply encode data into memory and/or retrieve and decode data from memory.

For ease of description, embodiments of the invention are described herein, for example, with reference to High Efficiency Video Coding (HEVC) or Universal Video Coding (VVC) reference software, the next generation video coding standard developed by the Video Coding Collaborative (JCT-VC) Group. the ITU-T Video Coding Experts (VCEG); and the ISO/IEC Moving Picture Experts Group (MPEG). One skilled in the art will appreciate that embodiments of the invention are not limited to HEVC or VVC.

Encoder and encoding method

Fig. 2 shows a schematic block diagram of an exemplary video encoder 20 that is configured to implement the methods of the present application. In the example of FIG. 2, video encoder 20 includes an input 201 (or input interface 201), a residual calculation block 204, a transform processing block 206, a quantization block 208, an inverse quantization block 210, and an inverse transform processing block 212, a reconstruction block 214, a contour block 220 filter, decoded picture buffer (DPB) 230, mode selector 260, entropy coding block 270, and output 272 (or output interface 272). The mode selector 260 may include an inter-predictor 244, an intra-predictor 254, and a splitter 262. Inter prediction block 244 may include a motion estimator and a motion compensation block (not shown). The video encoder 20 shown in FIG. 2 may also be referred to as a hybrid video encoder or a video encoder according to a hybrid video codec.

Residual calculator 204, transform processor 206, quantizer 208, mode selector 260 may be referred to as forming the forward signal path of the encoder 20, while inverse quantizer 210, inverse transform processor 212, reconstruction 214, buffer 216, loop filter 220, a decoded picture buffer (DPB) 230, inter predictor 244, and intra predictor 254 may be referred to as generating the reverse signal path of video encoder 20, with the reverse signal path of video encoder 20 corresponding to the signal path of the decoder (see video decoder 30 in FIG. 3). The inverse quantization block 210, the inverse transform processing block 212, the reconstruction block 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction block 244, and the intra prediction block 254 are also related to the formation of the "embedded decoder" of the video encoder 20.

Images and Image Sharing (Images and Blocks)

The encoder 20 may be configured to receive, for example, via input 201 an image 17 (or image data 17), for example an image of a sequence of images constituting a video or video sequence. The received image or image data may also be a pre-processed image 19 (or pre-processed image data 19). For simplicity, the following description refers to picture 17. Picture 17 may also be referred to as the current picture or the picture to be encoded (particularly in video encoding, to distinguish the current picture from other pictures, such as previously encoded and/or decoded pictures of the same video sequence, that is, a video sequence that also contains the current image).

A (digital) image is, or can be viewed as, a two-dimensional array or matrix of samples with intensity values. A sample in an array can also be called a pixel (short for picture element) or a pixel. The number of samples in the horizontal and vertical direction (or axis) of the array or image determines the size and/or resolution of the image. Three color components are typically used to represent a color, i.e. an image can be represented or include three arrays of samples. In RGB format or color space, an image contains a corresponding array of red, green, and blue color samples. However, in video encoding, each pixel is typically represented in a luma and chrominance format or color space, such as YCbCr, which contains a luminance component labeled Y (sometimes L is also used instead) and two chrominance components labeled Cb and Cr. The Y component of luminance (or luminance for short) represents the brightness or intensity of the gray level (eg, as in a grayscale image), while the two chrominance (or chroma for short) components Cb and Cr represent chrominance or color information components. Accordingly, an image in YCbCr format contains an array of luminance samples of values (Y) of luminance samples and two arrays of chrominance samples of chrominance values (Cb and Cr). RGB images can be converted or converted to YCbCr format and vice versa, this process is also known as color conversion or conversion. If the image is monochrome, it can only contain an array of brightness samples. Accordingly, the image may be, for example, an array of luminance samples in monochrome format, or an array of luminance samples and two corresponding arrays of chrominance samples in 4:2:0, 4:2:2 and 4:4:4 color format.

Embodiments of video encoder 20 may comprise an image splitter (not shown in FIG. 2) configured to split image 17 into multiple (generally non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264/AVC), or coding tree blocks (CTBs) or coding tree units (CTUs) (H.265/HEVC and VVC). The image dividing block may be configured to use the same block size for all images of a video sequence and a corresponding block size grid, or to change the block size between images or subsets or groups of images and divide each image into appropriate blocks.

In additional embodiments, the video encoder may be configured to directly receive image block 203 17, such as one, more or all of the blocks that make up image 17. Image block 203 may also be referred to as a current image block or an image block to be encoded.

Like image 17, image block 203 is again, or can be viewed as, a two-dimensional array or matrix of samples with intensity values (sample values), albeit smaller than image 17. In other words, block 203 may contain, for example, one array of samples (for example , a luma array in the case of a monochrome image 17 or a luminance or chroma array in the case of a color image) or three arrays of samples (for example, luminance and two chrominance arrays in the case of a color image 17) or any other number and/or type of arrays depending on the applied color format. The number of samples in the horizontal and vertical direction (or axis) of block 203 defines the size of block 203. Accordingly, a block may be, for example, an array of MxN discrete samples (M-column per N-row) or an array of MxN transform coefficients.

Embodiments of video encoder 20, as shown in FIG. 2 may be configured to encode picture 17 block by block, for example, encoding and prediction are performed for each block 203.

Remainder Calculation

Residual block 204 may be configured to calculate a residual block 205 (also referred to as residual 205) based on image block 203 and prediction block 265 (more details about predictor 265 will be provided later), for example, by subtracting the sample values of predictor 265 from the sample values. image block 203, sample by sample (pixel by pixel) to obtain a residual block 205 in the sample area.

transformation

The transform processing unit 206 may be configured to apply a transform, such as a discrete cosine transform (DCT) or a discrete sine transform (DST), to the sample values of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 may also be referred to as residual transform coefficients and represent the residual block 205 in the transform region.

Transform processing unit 206 may be configured to apply DCT/DST integer approximations such as the transforms specified for H.265/HEVC. Compared to the orthogonal DCT, such integer approximations are usually scaled by some factor. To preserve the norm of the residual block that is processed by forward and backward transformations, additional scaling factors are applied during the transformation. Scaling factors are usually chosen based on some constraints such as power of two scaling factors for shift operations, word length of the transform coefficients, trade-off between accuracy and implementation cost, etc. the inverse transform processor 212 (and the corresponding inverse transform, eg, the inverse transform processor 312 in video decoder 30) and the corresponding scaling factors for the forward transform, eg, transform processor 206, may be set in the encoder 20, respectively.

Embodiments of video encoder 20 (respectively, transform processing module 206) may be configured to output transform parameters, such as the type of transform or transforms, such as directly or encoded or compressed via entropy encoding unit 270, such that, for example, video decoder 30 may receive and use transform options for decoding.

Quantization

Quantizer 208 may be configured to quantize transform coefficients 207 to obtain quantized coefficients 209, for example by applying scalar quantization or vector quantization. The quantized coefficients 209 may also be referred to as quantized transform coefficients 209 or quantized residual coefficients 209.

The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207 . For example, an n-bit transform factor may be rounded to an m-bit transform factor during quantization, where n is greater than m. The degree of quantization can be changed by adjusting the quantization parameter (QP). For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, while larger quantization step sizes correspond to coarser quantization. The applicable quantization step size may be indicated by a quantization parameter (QP). The quantization parameter may, for example, be an index to a predetermined set of applicable quantization step sizes. For example, small quantization parameters may correspond to fine quantization (small quantization step sizes), and large quantization parameters may correspond to coarse quantization (large quantization step sizes), or vice versa. Quantization may include dividing by the quantization step size, and corresponding and/or inverse dequantization, for example by inverse quantizer 210, may include multiplying by the quantization step size. Embodiments according to some standards, such as HEVC, may be configured to use a quantization parameter to determine the quantization step size. Typically, the quantization step size can be calculated based on the quantization parameter using a fixed-point approximation of an equation involving division. Additional scaling factors can be introduced for quantization and dequantization to recover the residual block rate, which can be changed due to the scaling used in the fixed-point approximation of the equation for quantization step size and quantization parameter. In one exemplary implementation, inverse transform scaling and dequantization may be combined. Alternatively, customized quantization tables can be used and signaled from the encoder to the decoder, for example, in the bit stream. Quantization is a lossy operation in which the loss increases as the size of the quantization steps increases.

Embodiments of video encoder 20 (respectively, quantization module 208) may be configured to output quantization parameters (QPs), for example, directly or encoded via entropy encoding module 270, such that, for example, video decoder 30 may receive and apply quantization parameters for decoding.

Inverse quantization

Inverse quantizer 210 is configured to apply inverse quantization of quantizer 208 to quantized coefficients to obtain dequantized coefficients 211, for example, by applying an inverse quantization scheme applied by quantizer 208 based on or using the same quantization step as quantizer 208. The dequantized coefficients 211 may also be referred to as the dequantized residual coefficients 211 and correspond to, although typically not identical to, the transform coefficients due to quantization loss, to the transform coefficients 207.

Reverse transformation

The inverse transform processing unit 212 is configured to apply an inverse transform to the transform applied by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST) or other inverse transforms, to obtain a reconstructed residual block 213 (or corresponding dequantized coefficients 213) in the sample area. The recovered residual block 213 may also be referred to as a transformation block 213 .

Recovery

Reconstructor 214 (e.g., adder or adder 214) is configured to add transform block 213 (i.e., reconstructed residual block 213) to prediction block 265 to obtain reconstructed block 215 in the sample region, for example, by addition, sample by sample, the sample values of the reconstructed residual block 213 and the sample values of the prediction block 265 .

Filtration

The loop filter block 220 (or "loop filter" 220 for short) is configured to filter the reconstructed block 215 to obtain a filtered block 221, or more generally to filter the reconstructed samples to obtain filtered samples. The loop filter block, for example, is configured to smooth out pixel transitions or otherwise improve video quality. The loop filter block 220 may comprise one or more loop filters, such as a deblocking filter, an adaptive offset filter (SAO), or one or more other filters, such as a two-sided filter, an adaptive loop filter (ALF), sharpening, smoothing filters, or collaborative filters. filters, or any combination of them. Although the loop filter unit 220 is shown in FIG. 2 as a loop filter, in other configurations, the loop filter module 220 may be implemented as a post loop filter. The filtered block 221 may also be referred to as the filtered reconstructed block 221.

Embodiments of video encoder 20 (respectively, loop filter unit 220) may be configured to output loop filter parameters (such as adaptive sample offset information), for example, directly or encoded through entropy encoding unit 270, such that, for example, decoder 30 may receive and apply the same loop filter parameters or corresponding loop filters for decoding.

Decoded image buffer

Decoded Picture Buffer (DPB) 230 may be a memory that stores reference pictures or, more generally, reference picture data for encoding video data by video encoder 20. DPB 230 may be formed by any of a variety of memories such as dynamic random access memory. (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of storage devices. Decoded picture buffer (DPB) 230 may be configured to store one or more filtered blocks 221. Decoded picture buffer 230 may be further configured to store other previously filtered blocks, such as previously restored and filtered blocks 221, the same current picture, or different images, such as previously reconstructed images, and may provide complete previously reconstructed, i.e., decoded, images (and corresponding reference blocks and samples) and/or partially reconstructed current image (and corresponding reference blocks and samples), for example, for inter prediction . The decoded picture buffer (DPB) 230 may also be configured to store one or more unfiltered reconstructed blocks 215, or, in general, unfiltered reconstructed samples, for example, if the reconstructed block 215 is not filtered by the loop filtering block 220, or any other further processed version of the reconstructed blocks. or samples.

Mode selection (separation and prediction)

The mode selection unit 260 comprises a splitter 262, an inter prediction unit 244, and an intra prediction unit 254, and is configured to receive or obtain original image data, such as the original block 203 (the current block 203 of the current image 17), and reconstructed image data, such as filtered and/or or unfiltered reconstructed samples or blocks of the same (current) picture and/or from one or more previously decoded pictures, eg decoded picture buffer 230 or other buffers (eg string buffer, not shown). The reconstructed image data is used as reference image data for prediction, such as inter prediction or intra prediction, to obtain predictor 265 or predictor 265.

Mode selector 260 may be configured to determine or select a split for the current block prediction mode (including no split) and prediction mode (e.g., intra or inter prediction mode) and generate a corresponding prediction block 265 that is used to calculate the residual block. 205 and to restore the restored block 215.

Embodiments of mode selector 260 may be configured to select a split and prediction mode (eg, from those supported or available to mode selector 260) that provides the best match, or in other words, minimum residual (minimum residual means the best compression for transmission or storage), or minimal signaling overhead (minimum signaling overhead means better compression for transmission or storage), or which takes into account or balances both. The mode selector 260 may be configured to determine the division and prediction mode based on Rate Distortion Optimization (RDO), ie, select a prediction mode that provides minimal rate distortion. Terms such as "best", "minimum", "optimal", etc. in this context do not necessarily refer to the general "best", "minimum", "optimal", etc., but may also refer to fulfilling a termination or selection criterion, such as a value above or below a threshold, or other constraints potentially leading to a "sub-optimal selection" but reducing complexity and processing time.

In other words, partitioner 262 can be configured to partition block 203 into smaller partitions or subblocks (which form blocks again), for example, iteratively using quadrilateral partitioning (QT), binary partitioning (BT), or ternary tree partitioning (TT) or any combination thereof, and to perform, for example, prediction for each of the partitions or sub-blocks of the block, wherein the mode selection comprises selecting a tree structure of the partitioned block 203, and the prediction modes are applied to each of the partitions or sub-blocks of the block.

The following explains in more detail the separation (for example, by the separation unit 260) and the prediction processing (by the inter prediction unit 244 and the intra prediction unit 254) performed by the exemplary video encoder 20.

Separation

Divider 262 may divide (or break) the current block 203 into smaller pieces, such as smaller blocks of square or rectangular size. These smaller blocks (which may also be referred to as sub-blocks) can be further subdivided into even smaller sections. This is also called tree splitting or hierarchical tree splitting, in which a root box, e.g. at level 0 of the root tree (hierarchy level 0, depth 0), can be recursively split, e.g. split into two or more boxes of the next lower level of the tree, e.g. , nodes at tree level 1 (hierarchy level 1, depth 1), whereby these blocks can again be split into two or more next lower level blocks, e.g. tree level 2 (hierarchy level 2, depth 2), etc. until the split is complete, for example because a termination criterion is met, such as the maximum tree depth or minimum block size has been reached. Blocks that are not split further are also called leaf blocks or leaf nodes of the tree. A tree using two partitions is called a binary tree (BT), a tree using three partitions is called a ternary tree (TT), and a tree using four partitions is called a square tree (QT).

As mentioned earlier, the term "block" as used herein can be a part, in particular a square or rectangular part of an image. With respect to, for example, HEVC and VVC, a block may be either a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), and a transformation unit (TU) and/or corresponding blocks, such as a coding tree block (CTB). ), a coding block (CB), a transformation block (TB), or a prediction block (PB).

For example, a coding tree unit (CTU) can be either a luminance sample CTB, two corresponding chrominance sample CTBs of an image that has three sample arrays, or a monochrome image sample CTB or an image that is encoded using three separate color planes and the syntax structures used for coding samples. Accordingly, a coding tree block (CTB) may be a block of NxN samples for some value of N, such that dividing a component by CTB is a split. A coding unit (CU) can be either a luminance sample coding unit, two corresponding chrominance sample coding units of an image that has three sample arrays, or a monochrome image sample coding unit or an image that is encoded using three separate color planes and the syntax structures used for coding samples. Accordingly, a coding block (CB) may be a block of MxN samples for some values of M and N, so dividing the CTB into coding blocks is a division.

In embodiments, for example according to HEVC, a coding tree unit (CTU) may be partitioned into CUs using a square tree structure referred to as a coding tree. The decision as to whether to encode an image area using extrinsic (temporal) or intrinsic (spatial) prediction is made at the CU level. Each CU may be further partitioned into one, two or four PUs according to the partition type of the PUs. Within one PU, the same prediction process is applied and the corresponding information is transmitted to the decoder based on the PU. After obtaining the residual block by applying a prediction process based on the partition type of the PU, the CU may be divided into transform units (TUs) according to a different quadtree structure similar to the coding tree for the CU.

In embodiments, for example, according to the latest video coding standard currently being developed, called universal video coding (VVC), quad-binary tree (QTBT) partitioning is used to partition a coding block. In the block structure of QTBT, the CU can be square or rectangular. For example, a coding tree unit (CTU) is first divided by a square tree structure. The leaf nodes of the quadtree are further separated by a binary tree or a ternary (or ternary) tree structure. The leaf nodes of the partitioning tree are called coding units (CUs) and this segmentation is used for prediction and transformation processing without any further partitioning. This means that CU, PU and TU have the same block size in the QTBT coding block structure. In parallel with the block structure of QTBT, it has been proposed to use multiple partitioning, such as triple tree partitioning.

In one example, mode selector 260 of video encoder 20 may be configured to perform any combination of the separation methods described herein.

As described above, video encoder 20 is configured to determine or select the best or optimal prediction mode from a set of (predetermined) prediction modes. The set of prediction modes may include, for example, intra prediction modes and/or inter prediction modes.

Internal prediction

The intra prediction mode set may contain 35 different intra prediction modes, such as non-directional modes such as DC (or average) mode and planar mode, or directional modes, such as those defined in HEVC, or may comprise 67 different intra prediction modes, such as , non-directional modes such as DC (or medium) mode and planar mode, or directional modes such as those defined for VVC.

The intra prediction block 254 is configured to use the recovered samples of neighboring blocks of the same current picture to generate the intra prediction block 265 according to the intra prediction mode of the intra prediction mode set.

Block 254 intra prediction (or, in General, block 260 mode selection) is further configured to output parameters of intra prediction (or, in General, information indicating the selected intra prediction mode for the block) in block 270 entropy encoding in the form of elements 266 syntax for including the encoded image in the data 21 so that, for example, the video decoder 30 can receive and use the prediction parameters for decoding.

external prediction

The set (or possible) inter prediction modes depends on the available reference pictures (i.e., previous at least partially decoded pictures, such as those stored in DPB 230) and other inter prediction parameters, such as whether the reference picture is all or only a part of it, for example, the area of the search window around the area of the current reference image block is used to find the most suitable reference block and/or, for example, whether pixel interpolation is applied, for example, half/half pixel and/or quarter pixel interpolation or not.

In addition to the aforementioned prediction modes, a skip mode and/or a direct mode may be applied.

Inter prediction block 244 may include a motion estimator (ME) and a motion compensation (MC) block (both not shown in FIG. 2). The motion estimator may be configured to receive or obtain an image block 203 (current image block 203 for current image 17) and a decoded image 231, or at least one or a plurality of previously reconstructed blocks, e.g., reconstructed blocks of one or a plurality of other/different previously decoded images 231 for motion estimation. For example, the video sequence may comprise the current picture and previously decoded pictures 231, or in other words, the current picture and previously decoded pictures 231 may be part of or form a sequence of pictures that make up the video sequence.

Encoder 20 may, for example, be configured to select a reference block from a plurality of reference blocks of the same or different pictures from a plurality of other pictures and provide a reference picture (or reference picture index) and/or offset (spatial offset) between position (x coordinates , y) the reference block and the position of the current block as inter-prediction parameters for the motion estimation block. This displacement is also called motion vector (MV).

The motion compensation block is configured to obtain, for example, receive an inter prediction parameter and perform inter prediction based on or using the inter prediction parameter to obtain inter prediction block 265. The motion compensation performed by the motion compensator may include extracting or generating a prediction block based on the motion vector/block determined by the motion estimation, possibly performing sub-pixel-accurate interpolation. Interpolation filtering can generate additional pixel samples from known pixel samples, thus potentially increasing the number of candidate prediction blocks that can be used to encode an image block. Upon receiving the motion vector for the PU of the current picture block, the motion compensator may find the prediction block pointed to by the motion vector in one of the reference picture lists.

The motion compensation block may also generate syntax elements associated with blocks and a video slice for use by video decoder 30 when decoding image blocks of a video slice.

Entropy coding

Entropy encoding unit 270 is configured to apply, for example, an entropy encoding algorithm or scheme (for example, variable length coding (VLC) scheme, context adaptive VLC scheme (CAVLC), arithmetic coding scheme, binarization, context adaptive binary arithmetic coding (CABAC ) based on the syntax of context adaptive binary arithmetic coding (SBAC), probability interval entropy coding (PIPE), or other entropy coding technique or method) or traversal (without compression) of quantized coefficients 209, inter prediction parameters, intra prediction parameters, parameters loop filter and/or other syntax elements for obtaining encoded image data 21 that can be output via output 272, for example, in the form of a coded bitstream 21, so that, for example, video decoder 30 can receive and use parameters for decoding. The encoded bit stream 21 may be transmitted to video decoder 30 or stored in memory for later transmission or retrieval by video decoder 30.

Other structural variations of video encoder 20 may be used to encode the video stream. For example, non-transform-based encoder 20 may quantize the residual signal directly without transform processor 206 for some blocks or frames. In another implementation, encoder 20 may have quantizer 208 and inverse quantizer 210 combined into a single unit.

Decoder and decoding method

Fig. 3 shows an example of a video decoder 30 that is configured to implement the techniques of the present application. Video decoder 30 is configured to receive encoded picture data 21 (e.g., encoded bitstream 21), such as encoded by encoder 20, to obtain a decoded picture 331. The encoded picture data or bitstream contains information for decoding encoded picture data, such as data that represents blocks encoded video slice images and associated syntax elements.

In the example in FIG. 3, the decoder 30 includes an entropy decoding block 304, an inverse quantization block 310, an inverse transform processing block 312, a reconstruction block 314 (e.g., adder 314), a loop filter 320, a decoded picture buffer (DPB) 330, an inter prediction block 344, and a block 354 internal prediction. Inter prediction block 344 may be or include a motion compensation block. Video decoder 30 may, in some examples, perform a decoding step that is generally the reverse of the encoding step described with respect to video encoder 100 in FIG.

As explained with respect to the encoder 20, the inverse quantizer 210, the inverse transform processor 212, the reconstructor 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction unit 344 and the intra prediction unit 354 are also referred to as constituting an "embedded decoder » of the video encoder 20. Accordingly, the inverse quantizer 310 may be identical in function to the inverse quantizer 110, the inverse transform processing unit 312 may be identical in function to the inverse transform processing unit 212, the restorer 314 may be identical in function to the restorer 214, loop filter 320 may be identical in function to loop filter 220, and decoded image buffer 330 may be identical in function to decoded image buffer 230. Therefore, the explanations provided for the respective blocks and functions of the video encoder 20 apply to the respective modules and functions of the video decoder 30, respectively.

Entropy decoding

The entropy decoding module 304 is configured to parse the bit stream 21 or, in general, the encoded image data 21) and perform, for example, entropy decoding on the encoded image data 21 to obtain, for example, quantized coefficients 309 and/or decoded coding parameters ( not shown in FIG. 3), e.g., any or all of the inter prediction parameters (e.g., reference picture index and motion vector), inter prediction parameter (e.g., index or intra prediction mode), transform parameters, quantization parameters, loop filter parameters and/or other syntax elements. Entropy decoding unit 304 may be configured to apply decoding algorithms or schemes corresponding to coding schemes as described in relation to entropy encoding unit 270 of encoder 20. Entropy decoding unit 304 may be further configured to provide inter prediction parameters, an intra prediction parameter, and/or other syntax elements to mode select block 360; and other parameters to other blocks of decoder 30. Video decoder 30 may receive syntax elements at the video slice level and/or video block level.

Inverse quantization

An inverse quantizer 310 may be configured to receive quantization parameters (QPs) (or inverse quantization related information in general) and quantized coefficients from the encoded image data 21 (e.g., by parsing and/or decoding, e.g., by a block 304 entropy decoding) and apply, based on the quantization parameters, inverse quantization to the decoded quantized coefficients 309 to obtain dequantized coefficients 311, which may also be referred to as transform coefficients 311. The inverse quantization process may include using a quantization parameter determined by video encoder 20 for each video block in a video slice to determine the amount of quantization, and likewise, the amount of inverse quantization to be applied.

Reverse transformation

The inverse transform processor 312 may be configured to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and to apply a transform to the dequantized coefficients 311 to obtain reconstructed residual blocks 213 in the sample area. The recovered residual blocks 213 may also be referred to as transform blocks 313 . The transformation may be an inverse transformation, eg, inverse DCT, inverse DST, inverse integer transformation, or a conceptually similar inverse transformation process. The inverse transform processing unit 312 may be further configured to receive transform parameters or corresponding information from the encoded image data 21 (eg, by parsing and/or decoding, eg, by entropy decoding unit 304) to determine the transform to be applied to the dequantized coefficients. 311.

Recovery

Reconstructor 314 (e.g., adder or adder 314) may be configured to add recovered residual block 313 to predictor 365 to obtain reconstructed block 315 in the sample region, for example, by adding the sample values of the reconstructed residual block 313 to the sample values of predictor 365.

Filtration

The loop filter block 320 (either in the encoding loop or after the encoding loop) is configured to filter the reconstructed block 315 to obtain a filtered block 321, for example, to smooth pixel transitions or otherwise improve video quality. The loop filter block 320 may comprise one or more loop filters, such as a deblocking filter, an adaptive offset (SAO) filter, or one or more other filters, such as a two-sided filter, an adaptive loop filter (ALF), sharpening, smoothing filters, or collaborative filters. filters, or any combination of them. Although the loop filter module 320 is shown as a loop filter in FIG. 3, in other configurations, the loop filter module 320 may be implemented as a post loop filter.

Decoded image buffer

The decoded video blocks 321 of the picture are then stored in the decoded picture buffer 330, which stores the decoded pictures 331 as reference pictures for subsequent motion compensation for other pictures and/or for display output, respectively.

The decoder 30 is configured to output the decoded image 311, for example, via output 312 for presentation or viewing to a user.

Prediction

Inter prediction block 344 may be identical to inter prediction block 244 (in particular, motion compensation block), and intra prediction block 354 may be identical to inter function prediction block 254 and performs split or split decisions and prediction based on split parameters and/or prediction or corresponding information received from the encoded image data 21 (eg, by parsing and/or decoding, eg, by entropy decoding unit 304). Mode selector 360 may be configured to perform prediction (intra or inter prediction) for each block based on the reconstructed images, blocks, or corresponding samples (filtered or unfiltered) to obtain prediction block 365 .

When a video slice is encoded as an intra (I) slice, the intra prediction block 354 of the mode select block 360 is configured to generate a prediction block 365 for a picture block of the current video slice based on the signaled intra prediction mode and data from the previously decoded blocks of the current picture. When a video image is encoded as an inter-coded (i.e., B or P) slice, the inter prediction block 344 (e.g., motion compensation block) of the mode select block 360 is configured to create prediction blocks 365 for the current video slice video block based on motion vectors and other syntax elements. received from block 304 entropy decoding. For inter prediction, prediction blocks may be created from one of the reference pictures in one of the reference picture lists. Video decoder 30 may construct lists of key frames, List 0 and List 1, using default construction methods based on reference pictures stored in DPB 330.

The mode selector 360 is configured to determine prediction information for a video block of the current video slice by analyzing motion vectors and other syntax elements, and uses the prediction information to generate prediction blocks for the current video block being decoded. For example, the mode selector 360 uses some of the received syntax elements to determine the prediction mode (eg, intra or inter prediction) used to encode the video blocks of the video slice, the type of inter prediction slice (eg, B slice, P slice, or GPB slice). ), construction information for one or more reference picture lists for a slice, motion vectors for each intercoded video block of the slice, inter prediction status for each inter prediction video block of said slice, and other information for decoding video blocks in the current video slice.

Other variants of video decoder 30 may be used to decode encoded picture data 21. For example, decoder 30 may generate an output video stream without loop filter block 320 . For example, non-transform-based decoder 30 can directly inverse quantize the residual signal without inverse transform processing unit 312 for some blocks or frames. In another implementation, video decoder 30 may have an inverse quantizer 310 and an inverse transform processor 312 combined into one block.

It should be understood that in encoder 20 and decoder 30, the processing result of the current step may be further processed and then output to the next step. For example, after interpolation filtering, motion vector output, or loop filtering, an additional operation such as trimming or shifting may be performed on the result of the interpolation filtering, motion vector output, or loop filtering processing.

It should be noted that additional operations can be applied to derived motion vectors of the current block (including, but not limited to, affine mode checkpoint motion vectors, sub-block motion vectors in affine, planar, ATMVP modes, temporal motion vectors, etc. .). For example, the motion vector value is limited to a predefined range according to its representation bit. If the representing bit of the motion vector is bitDepth, then the range is -2^(bitDepth-1)~2^(bitDepth-1)-1, where "^" means exponentiation. For example, if bitDepth is set to 16, the range is -32768 ~ 32767; if bitDepth is set to 18, the range is -131072 ~ 131071. For example, the motion vector derivative value (for example, the MV of four 4×4 sub-blocks in one 8×8 block) is limited such that the maximum difference between the integer parts of the four MVs of a 4× sub-block 4 does not exceed N pixels, for example, not more than 1 pixel. Here are two ways to limit the motion vector according to bitDepth.

Method 1: Remove the overflow MSB (most significant bit) using streaming operations

ux= ( mvx+2 ^bitDepth ) % 2 ^bitDepth (1)

mvx = ( ux >= 2 ^bitDepth-1 ) ? (ux − 2 ^bitDepth ) : ux(2)

uy= ( mvy+2 ^bitDepth ) % 2 ^bitDepth (3)

mvy = ( uy >= 2 ^bitDepth-1 ) ? (uy − 2 ^bitDepth ) : uy (4)

where mvx is the horizontal component of the motion vector of the image block or sub-block, mvy is the vertical component of the motion vector of the image block or sub-block, and ux and uy indicate an intermediate value;

For example, if the value of mvx is -32769, after applying formulas (1) and (2), the resulting value will be 32767. In a computer system, decimal numbers are stored as two's complement. Two's complement -32769 is 1.0111,1111,1111,1111 (17 bits), then the most significant bit is discarded, so the resulting two's complement is 0111,1111,1111,1111 (32767 decimal), which is the same as yield by applying formulas (1) and (2).

ux= ( mvpx + mvdx +2 ^bitDepth ) % 2 ^bitDepth (5)

mvx = ( ux >= 2 ^bitDepth-1 ) ? (ux − 2 ^bitDepth ) : ux(6)

uy= ( mvpy + mvdy +2 ^bitDepth ) % 2 ^bitDepth (7)

mvy = ( uy >= 2 ^bitDepth-1 ) ? (uy − 2 ^bitDepth ) : uy (8)

Operations can be applied during the summation of mvp and mvd, as shown in formulas (5) - (8).

Method 2: Remove the most significant bit of the overflow by trimming the value

vx = Clip3(-2 ^bitDepth-1 , 2 ^bitDepth-1 -1, vx)

vy = Clip3(-2 ^bitDepth-1 , 2 ^bitDepth-1 -1, vy)

where vx is the horizontal component of the motion vector of the image block or subblock, vy is the vertical component of the motion vector of the image block or subblock; x, y, and z respectively correspond to the three input values of the MV clipping process, and the Clip3 function definition is as follows:

Clip3 (x, y, z) = x; z<xy;

(otherwise - otherwise)

Fig. 4 is a schematic diagram of a video encoding apparatus 400 according to an embodiment of the disclosure. The video encoding device 400 is suitable for implementing the disclosed embodiments as described herein. In an embodiment, video encoding device 400 may be a decoder, such as video decoder 30 of FIG. 1A or an encoder such as video encoder 20 of FIG. 1A.

The video encoding device 400 includes input ports 410 (or input ports 410) and receiver (Rx) units 420 for receiving data; a processor, logical unit, or central processing unit (CPU) 430 for processing data; transmitter units (Tx) 440 and output ports 450 (or output ports 450) for data transmission; and a memory 460 for storing data. The video encoder 400 may also include optical-electrical (OE) components and electrical-optical (EO) components connected to input ports 410, receiver units 420, transmitter units 440, and output ports 450 for output or input of optical or electrical signals.

The processor 430 is implemented in hardware and software. Processor 430 may be implemented as one or more CPU chips, cores (eg, as a multi-core processor), FPGA, ASIC, and DSP. Processor 430 communicates with input ports 410, receiver units 420, transmitter units 440, output ports 450, and memory 460. Processor 430 includes an encoder 470. Encoder 470 implements the disclosed embodiments described above. For example, encoding unit 470 implements, processes, prepares, or provides various encoding operations. Thus, the inclusion of the encoding unit 470 provides a significant improvement in the functionality of the video encoding device 400 and allows the video encoding device 400 to be converted to another state. Alternatively, encoder 470 is implemented as instructions stored in memory 460 and executed by processor 430.

Memory 460 may comprise one or more disks, tape drives, and solid state drives and may be used as a storage device for overflowing data, for storing programs when such programs are selected for execution, and for storing instructions and data that are read during program execution. Memory 460 may be, for example, volatile and/or nonvolatile and may be read only memory (ROM), random access memory (RAM), content-addressable ternary memory (TCAM), and/or static random access memory (SRAM).

Fig. 5 is a simplified block diagram of a device 500 that can be used as one or both of source device 12 and target device 14 in FIG. 1A according to an exemplary embodiment .

Processor 502 in device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or a plurality of devices capable of manipulating or processing information that currently exists or will be developed in the future. Although the disclosed implementations may be implemented with a single processor, as illustrated by processor 502, for example, speed and efficiency advantages may be achieved using more than one processor.

Memory 504 in device 500 may be read only memory (ROM) or random access memory (RAM) in some implementation. The memory 504 may be any other suitable type of storage device. Memory 504 may include code and data 506 accessed by processor 502 using bus 512. Memory 504 may further include an operating system 508 and application programs 510, application programs 510 include at least one program that allows processor 502 to perform the methods described here. For example, applications 510 may include applications 1 through N, which further include a video encoding application that performs the methods described herein.

The device 500 may also include one or more output devices such as a display 518. The display 518 may be, in one example, a touch-sensitive display that combines the display with a touch-sensitive element capable of reading touch inputs. Display 518 may be connected to processor 502 via bus 512.

Although shown here as a single bus, bus 512 of device 500 may be comprised of multiple buses. In addition, secondary storage 514 may be directly connected to other components of the device 500 or may be accessed via a network and may contain one integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Thus, device 500 can be implemented in a wide variety of configurations.

Intra-prediction of the chrominance samples may be performed using the recovered luminance block samples.

During the development of HEVC, cross-component linear model (CCLM) intra-chrominance prediction was proposed [J. Kim, S.-W. Park, J.-J. Park, B.-M. Cheon, Intra Chroma Prediction Using Inter Channel Correlation, document JCTVC-B021, Jul. 2010]. CCLM uses a linear correlation between a chrominance sample and a luma sample at the corresponding position in the coding block. When a chrominance block is encoded using CCLM, a linear model is derived from the reconstructed adjacent luma and chrominance samples using linear regression. The chrominance samples in the current block can then be predicted from the reconstructed luminance samples in the current block with the derived linear model (see FIG. 6):

,

,

и

определяются способом наименьших квадратов следующим образом:where C and L indicate the chrominance and lightness values, respectively. Parameters

and

are defined by the least squares method as follows:

,

,

where M(A) is the mean of A and R(A, B) is defined as follows:

.

.

If the encoded or decoded image has a format that specifies a different number of samples for the luminance and chrominance components (eg, YCbCr 4:2:0 format), the luminance samples are downsampled before modeling and prediction.

The method has been adopted for use in VTM2.0. In particular, the derivation of parameters is done as follows:

,

,

,

,

whereL(n) represents the top and left adjacent reconstructed reduced brightness samples sampling, C(n) represents the top and left adjacent reconstructed chrominance samples.

In [G. Laroche, J. Taquet, C. Giske, P. Onno (Canon), “CE3: Cross-component linear model simplification (Test 5.1)”, Input document to 12th JVET Meeting in Macao, China, Oct. 2018] another method was proposed for obtaining αandβ (see Fig. 7). In particular, the parameters α and β of the linear model are obtained according to the following equations:

и

- положения максимального и минимального значений в выборках яркости.where

and

- positions of the maximum and minimum values in the brightness samples.

It has also been proposed to implement the division operation by multiplying by a number, which is stored in a look-up table (LUT), which is indicated in table 1. This replacement is possible in the following way:

параметр сдвига, задающий точность.where

a shift parameter that specifies the precision.

можно рассчитать следующим образом:Table 1 shows the mapping of the LUT index range (specified in the first row of the table) to the lists of values stored in the LUT, with each list corresponding to its index range. It can be seen that the values

can be calculated as follows:

.

.

выполняется следующим образом:Using the LUT defined in Table 1 (or calculated equivalently using the equation above), the calculation

is done as follows:

можно разложить на несколько частей, т. е.

, потому что значение α используется в дополнительном вычислении. Это разложение обеспечивает гибкость точности вычислений на различных этапах, и, таким образом, можно перераспределять битовую глубину операций умножения по шагам получения значения предсказанной выборки цветности. В конкретной реализации:Shift parameter

can be decomposed into several parts, i.e.

, because the value of α is used in the extra calculation. This decomposition provides flexibility in the accuracy of the calculations at different stages, and thus it is possible to redistribute the bit depth of the multiplication operations over the steps of obtaining the value of the predicted chrominance sample. In a specific implementation:

.

.

Table 1. An example LUT for implementing a division operation using multiplication by a number

LUT index ranges: 0 32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 31 63 95 127 159 191 223 255 287 319 351 383 415 447 479 511 Values stored in LUT: 65536 1985 1008 675 508 407 339 291 255 226 204 185 170 157 145 136 32768 1927 992 668 504 404 337 289 254 225 203 185 169 156 145 135 21845 1872 978 661 500 402 336 288 253 225 202 184 169 156 145 135 16384 1820 963 655 496 399 334 287 252 224 202 184 168 156 144 135 13107 1771 949 648 492 397 332 286 251 223 201 183 168 155 144 135 10922 1724 936 642 489 394 330 284 250 222 201 183 168 155 144 134 9362 1680 923 636 485 392 329 283 249 222 200 182 167 154 144 134 8192 1638 910 630 481 390 327 282 248 221 199 182 167 154 143 134 7281 1598 897 624 478 387 326 281 247 220 199 181 166 154 143 134 6553 1560 885 618 474 385 324 280 246 219 198 181 166 153 143 133 5957 1524 873 612 471 383 322 278 245 219 197 180 165 153 142 133 5461 1489 862 606 468 381 321 277 244 218 197 180 165 153 142 133 5041 1456 851 601 464 378 319 276 243 217 196 179 165 152 142 132 4681 1424 840 595 461 376 318 275 242 217 196 179 164 152 141 132 4369 1394 829 590 458 374 316 274 241 216 195 178 164 152 141 132 4096 1365 819 585 455 372 315 273 240 215 195 178 163 151 141 132 3855 1337 809 579 451 370 313 271 240 214 194 177 163 151 140 131 3640 1310 799 574 448 368 312 270 239 214 193 177 163 151 140 131 3449 1285 789 569 445 366 310 269 238 213 193 176 162 150 140 131 3276 1260 780 564 442 364 309 268 237 212 192 176 162 150 140 131 3120 1236 771 560 439 362 307 267 236 212 192 175 161 149 139 130 2978 1213 762 555 436 360 306 266 235 211 191 175 161 149 139 130 2849 1191 753 550 434 358 304 265 234 210 191 174 161 149 139 130 2730 1170 744 546 431 356 303 264 234 210 190 174 160 148 138 130 2621 1149 736 541 428 354 302 263 233 209 189 173 160 148 138 129 2520 1129 728 537 425 352 300 262 232 208 189 173 159 148 138 129 2427 1110 720 532 422 350 299 261 231 208 188 172 159 147 137 129 2340 1092 712 528 420 348 297 260 230 207 188 172 159 147 137 129 2259 1074 704 524 417 346 296 259 229 206 187 172 158 147 137 128 2184 1057 697 520 414 344 295 258 229 206 187 171 158 146 137 128 2114 1040 689 516 412 343 293 257 228 205 186 171 157 146 136 128 2048 1024 682 512 409 341 292 256 227 204 186 170 157 146 136 128

линейной модели имеет целочисленное представление с фиксированной точкой дробного значения, точность

определяется значением S1=S-S0, которое используется для получения значения предсказанной выборки цветности:In this case, the coefficient

linear model has an integer representation with a fixed point of a fractional value, precision

is defined by the value S1=S-S0, which is used to obtain the value of the predicted chrominance sample:

The LUT size is quite critical for the hardware implementation of the encoder or decoder. The easiest way to solve this problem is to regularly downsample the LUT, keeping only every Nth element (here N is the downsampling factor) of the original given LUT, i.e. in table 1.

, извлечение из LUT определяется по-другому:After regular subsampling (subsampling) with a power-of-two subsampling factor equal to

, extraction from the LUT is defined differently:

instead of

будет иметь небольшое значение, больше, чем вероятность того, что эта разница будет большой. Другими словами, вероятность появления значения из таблицы 1 уменьшается от левого столбца к правому столбцу, а внутри каждого столбца эта вероятность уменьшается от первого до последнего элемента, принадлежащего этому столбцу. Примерная зависимость вероятности возникновения от значения

приведена на рисунке 8.It can be seen that for natural images the probability

will have a small value, more than the probability that this difference will be large. In other words, the probability of occurrence of a value from table 1 decreases from the left column to the right column, and within each column, this probability decreases from the first to the last element belonging to this column. Approximate dependence of the probability of occurrence on the value

shown in Figure 8.

Therefore, keeping only every Nth element of the original LUT is far from optimal, since it corresponds to an equiprobable distribution of its argument, which is not true.

Considering this distribution, a better compromise between LUT size and computational accuracy can be achieved than conventional subsampling can achieve.

, и значение этого шага увеличивается с индексом записи.It is proposed to define the LUT using non-linear indexing, in particular, in such a way that two adjacent LUT entries correspond to different steps

, and the value of this step increases with the index of the entry.

. Положение старшего бита

определяет точность (т. е. шаг между двумя соседними записями LUT) на основе положения старших битов. Большие значения положения старшего бита соответствуют меньшей точности и большему значению шага.One computationally efficient solution is to determine the index in the LUT using the few most significant bits

. MSB position

determines the precision (i.e., the step between two adjacent LUT entries) based on the position of the most significant bits. Large values of the MSB position correspond to less precision and a larger step value.

.A specific embodiment is shown in Fig.9 and Fig.10. Figure 9 shows a flowchart for calculating LUT values, and Figure 10 shows how to get the index "idx" in the LUT corresponding to the input value

.

. Можно заметить, что на самом деле LUT имеет поддиапазоны, каждый поддиапазон содержит записи «col_max». Внутри поддиапазона значение «ctr» увеличивается с равным значением «шага». Для поддиапазона, имеющего индекс «lut_shift+1», который больше 1, соответствующее значение «шага» увеличено вдвое по сравнению с поддиапазоном «lut_shift». Первые два поддиапазона будут иметь шаг, равный единице из-за порога в операции «step=1<<max(0,lut_shift)».Using the steps shown in Figure 9, values can be obtained that will be stored in the LUT for later use in the CCLM simulation. On fig. 9 variable ctr iterates over all possible values

. You can see that the LUT actually has subranges, each subrange contains "col _{max" entries.} Within a subrange, the "ctr" value is incremented by an equal "step" value. For a subband having an index "lut_shift+1" that is greater than 1, the corresponding "step" value is doubled compared to the subrange "lut_shift". The first two subranges will have a step equal to one due to the threshold in the "step=1<<max(0,lut_shift)" operation.

As shown in FIG. 9, the flowchart illustrates an exemplary process for creating a lookup table in accordance with an embodiment of the invention. At block 902, the video encoder starts a lookup table generation process. The video encoding device may be a decoder, such as video decoder 30 in FIG. 1A-1B, FIG. 3 or an encoder such as video encoder 20 in FIG. 1A-1B, FIG. 2 or the video encoding device 400 of FIG. 4 or device 500 of FIG. five.

At step 904, let ctr=1 and lut_shift=1. At block 906, it is determined if the index lut_shift< lut_shiftmax. If lut_shift< lut_shiftmax, the step is calculated as 1<<max(0, lut_shift), and col=0 in step 908; otherwise, the generation process ends at step 922. At step 910, the initial offset is provided by step "ctr=ctr+(step>>1)". Then whether col<colmax is determined in step 912. If col<colmax, LUT[ctr]=(1<<S)/ctr in step 914, here, the precomputed LUT values that are determined in steps 912-918 generate one string LUT; otherwise, let index lut_shift= lut_shift+1 in step 920. In step 916, ctr values corresponding to the start of each subband are set to ctrl+step. At block 918, the generation process proceeds to the next column, then the process returns to block 912.

An exemplary LUT generated using the block diagram shown in Fig. 9 is shown in Table 2. A row in this table corresponds to a subband with the index "lut_shift". This table was obtained with "lut_shift_max",equal to 6, and "col_max", equal to 3, resulting in 48 entries in the LUT. The ctr values corresponding to the start of each sub-range (zero "col" value) are as follows:

0, 8, 17, 35, 71, 143, 287.

These values are not always a power of two because each subband is subsampled relative to the middle of the subband. The corresponding initial offset is provided by the step "ctr=ctr+(step>>1)" shown in fig. 9. The "step" value for "lut_shift" values not greater than 0 is set to 1.

Table 2. An example LUT generated using the block diagram shown in Figure 2. nine

lut_shift col 0 one 2 3 4 five 6 7 -one 65536 32768 21845 16384 13107 10922 9362 8192 0 7281 6553 5957 5461 5041 4681 4369 4096 one 3640 3276 2978 2730 2520 2340 2184 2048 2 1820 1638 1489 1365 1260 1170 1092 1024 3 910 819 744 682 630 585 546 512 4 455 409 372 341 315 292 273 256 five 227 204 186 170 157 146 136 128

обрабатывается для определения положения соответствующей записи в LUT, показанной в таблице 2. Старший ненулевой бит входного значения помечается как 1002. Положение этого бита определяет значение «msb». Фактически, значение msb - это log2 () входного значения 1001. Вычитание 1 из «msb» дает значение «lut_shift», которое выбирает строку в таблице 2. В противном случае шаг рассчитывается как «1<<lut_shift».In FIG. 10A is a binary representation 1001 of an input value (for example, the difference

processed to determine the position of the corresponding entry in the LUT shown in Table 2. The most significant non-zero bit of the input value is marked 1002. The position of this bit determines the value of "msb". In fact, the value of msb is the log2() of the input value 1001. Subtracting 1 from "msb" gives the value "lut_shift" which selects a row in table 2. Otherwise, the step is calculated as "1<<lut_shift".

Column selection is done by taking the "col _max" bits next to 1002. The value of the "col" column in table 2 is obtained as follows:

the value "high_bits" is obtained by selecting the bits "col _max+ 1" following the high bit 1020;

col is set to the value stored in "high_bits" minus 1.

For the case where the alignment step "ctr=ctr+(step>>1)" is not performed in Fig. 9, the output of the "lut_shift" value is the same, and the output of the "col" value is even simpler (Fig. 10B): the value of "col" obtained by choosing bits "col_max'' following the MSB 1020. This index derivation method corresponds to Table 3. The 'ctr' values (FIG. 9) corresponding to the beginning of each sub-band (a 'col' value of zero) are as follows:

0, 8, 16, 32, 64, 128, 256

Table 3. Another exemplary LUT generated using the block diagram shown in FIG. 9, in case "ctr=ctr+(step>>1)" is omitted

lut_shift col 0 one 2 3 4 five 6 7 -one 65536 32768 21845 16384 13107 10922 9362 8192 0 7281 6553 5957 5461 5041 4681 4369 4096 one 3855 3449 3120 2849 2621 2427 2259 2114 2 1985 1771 1598 1456 1337 1236 1149 1074 3 1008 897 809 736 675 624 579 541 4 508 451 407 370 339 313 291 271 five 255 226 204 185 170 157 145 136

When inferring the value "col", msb can be less than or equal to "col _max ". _In this case, the value of "col" is set to the least significant bits _{of "} col _max " of the input difference 1001.

In practical implementations, the LUT index is one-dimensional. It is generally understood that the LUTs shown in Tables 2 and 3 can be addressed linearly using a set of indices equal to:

.

.

Both LUTs shown in Table 2 and Table 3 store values that vary greatly in magnitude. Therefore, it makes sense to have the same number of bits for all values stored in the LUT. The value retrieved from the LUT can be further shifted to the left according to the value of lut_shift. The only exception to this rule is the first four values, which have excellent precision with the last four values in that row. However, this problem can be solved with an additional LUT that stores these additional offsets for the first four values. In this embodiment, the multiplier value is restored from the value retrieved from the LUT as follows:

. Значение δ устанавливается равным 3, 2, 1, 1, соответственно, для значений «idx», меньших или равных 4. Справочная таблица для этого варианта приведена в таблице 4.where

. The value of δ is set to 3, 2, 1, 1, respectively, for "idx" values less than or equal to 4. A lookup table for this option is given in Table 4.

Table 4. Another exemplary LUT generated using the block diagram shown in Fig. 9 in case of equal accuracy within the ranges.

lut_shift col 0 one 2 3 4 five 6 7 -one 128 128 171 128 205 171 146 128 0 228 205 186 171 158 146 137 128 one 241 216 195 178 164 152 141 132 2 248 221 200 182 167 155 144 134 3 252 224 202 184 169 156 145 135 4 254 226 204 185 170 157 146 136 five 128 128 171 128 205 171 146 128

It can be seen that the last rows of Table 4 are very similar to each other. Therefore, it is possible to reduce the size of the LUT by keeping only one row for some sets of subranges. In specific embodiments, when the value of "lut_shift" is greater than a certain threshold, the value of lut_shift is set equal to the threshold value, and the value of δ is reduced by the difference between the initial value of "lut_shift" and the value of the threshold value.

Fig. 11 is a block diagram illustrating exemplary block chroma sample intra prediction using a cross-component linear model. At 1102, the video encoder obtains the recovered luminance samples. The video encoding device may be a decoder, such as video decoder 30 in FIG. 1A-1B, FIG. 3 or an encoder such as video encoder 20 in FIG. 1A-1B, FIG. 2 or the video encoding device 400 of FIG. 4 or device 500 of FIG. five.

In step 1104, the video encoder determines the positions of the maximum and minimum values of the reconstructed sample in the reconstructed luma samples. For example, the reconstructed luma samples are neighboring reconstructed luminance samples corresponding to the chrominance sample.

In step 1106, the video encoder obtains the difference between the maximum and minimum of the reconstructed luminance samples.

In step 1108, the video encoder computes an index for the LUT to obtain a multiplier value that corresponds to the difference between the maximum and minimum of the reconstructed luminance samples.

For example, the video encoder determines the position of the MSB of the difference between the maximum and minimum luminance samples and uses the set of bits following the position of the MSB of the difference between the maximum and minimum luminance samples as an index for the LUT to extract the value. The position of the most significant bit of the difference between the maximum and minimum values of the brightness samples can be obtained as the base two logarithm of said difference. The video encoder determines the set of bits following the position of the MSB of the difference. As a possible result, the bitset includes four bits.

The LUT is generated with or without an alignment step. The LUT may include at least two adjacent values stored in the LUT that correspond to different steps of the received difference, and the value of this step increases with the value of the difference or is a constant.

As disclosed in exemplary tables 1-4, the LUT may include sub-ranges of values. The step of the value of the difference between the maximum and minimum of the reconstructed brightness samples is constant within the subrange; and the step for different subranges is different. For example, the value step of the difference between the maximum and minimum of the reconstructed luminance samples increases with increasing subband index. For example, the step value of the difference between the maximum and minimum of the reconstructed luminance samples may be a power of two of the subband index.

Let the LUT include at least three values: a first value, a second value, and a third value. Among the three values, the first value and the second value are two adjacent values, and the second value and the third value are two adjacent values. The step (ie, precision or difference) between the first value and the second value may be equal to or different from the step between the second value and the third value. If the first value is indexed by the first bit set and the second value is indexed by the second bit set, the first value is less than the second value when the value of the first bit set is greater than the value of the second bit set; or the first value is greater than the second value when the value of the first set of bits is less than the value of the second set of bits.

The LUT is divided into subbands. The subband index is determined using the position of the most significant non-zero bit of the difference between the maximum and minimum of the reconstructed luma samples. For example, the subrange size is 8 and the number of subranges is 6. As another example, different adjacent subranges have different increase in value with the same step.

The LUT may include non-linear indices. Two adjacent LUT entries correspond to different stepsL(B)-L(A), where L(B) represents the maximum value of recovered samples luminance, L(A) represents the minimum value of the reconstructed luminance samples. The write step value can increase as the write index increases.

When a LUT uses multiple MSBs L(B) -L(A), the position of the MSB L (B)-L(A) determines the precision (i.e., the step between two adjacent LUT entries) based on the position of the MSBs. Larger MSB positions may correspond to lower precision and larger increments.

In step 1110, the video encoder obtains the linear model parameters α and β by multiplying the extracted value by the difference between the maximum and minimum values of the reconstructed chrominance samples.

In step 1112, the video encoder calculates a predicted chrominance sample value using the obtained linear model parameters α and β.

Fig. 12 is a block diagram showing an exemplary structure of an apparatus 1200 for intra prediction of a block chrominance sample by applying a cross-component linear model. Device 1200 is configured to perform the above methods and may include:

an acquisition unit 1210, configured to obtain the reconstructed luminance samples;

a determiner 1220, configured to determine maximum and minimum luminance sample values based on the reconstructed luminance samples;

an acquisition unit 1210, further configured to obtain a difference between the maximum and minimum values of the luminance samples.

a determiner 1220, further configured to determine the position of the most significant bit of the difference between the maximum and minimum values of the brightness samples. For example, the position of the most significant bit of the difference between the maximum and minimum luminance sample values is the base two logarithm of said difference. As an implementation, the most significant bit is the first non-zero bit.

The apparatus 1200 further includes a calculator 1230 configured to extract a value from a look-up table (LUT) using a set of bits as an index, the set of bits following the position of the most significant bit of the difference between the maximum and minimum values of the brightness samples, obtaining the parameters α and β of a linear models based on the selected value; and calculating a predicted chrominance sample value using the obtained linear model parameters α and β. For example, a bitset contains four bits.

The calculation unit may derive the parameters α and β of the linear model based on the selected value and the difference between the maximum and minimum values of the reconstructed chrominance samples. For example, the calculation block obtains the parameters α and β of the linear model by multiplying the extracted value by the difference between the maximum and minimum values of the reconstructed chrominance samples.

Advantage of Embodiments

1. The index for the LUT is calculated in an elegant way that extracts a few bits in the binary representation. As a result, the efficiency of extracting the value from the LUT is improved.

2. As the efficiency of extracting a value from the LUT increases, the value of the multiplier for obtaining the parameters α and β of the linear model is reduced to a minimum.

3. LUT size is minimized. The division function curve (f (x) = 1/x, known as a hyperbola) in the embodiments of the present invention has been approximated as follows:

1) LUT table size can be 16

(minimum number of entries to fit a 1/x curve whose derivative varies from 0 to infinity)

2) LUT elements have a non-linear dependence on the record index

(for 1/x curve fitting)

3) Multipliers (LUT elements) are 3-bit unsigned integers (0…7).

(minimum precision to fit a 1/x curve whose derivative varies from 0 to infinity)

The following is an explanation of the applications of the encoding method as well as the decoding method as shown in the above embodiments, and the system using them.

Fig. 13 is a block diagram showing a content supply system 3100 for realizing a content distribution service. This content delivery system 3100 includes a capture device 3102, a terminal device 3106, and optionally includes a display 3126. The capture device 3102 communicates with the terminal device 3106 over a communication line 3104. The communication link may include the communication channel 13 described above. Link 3104 includes, but is not limited to, WIFI, Ethernet, cable, wireless (3G/4G/5G), USB, or any combination thereof, and the like.

The capture device 3102 generates data and may encode the data in an encoding manner as shown in the above embodiments. Alternatively, the capture device 3102 may distribute the data to a streaming server (not shown in the figures) and the server encodes the data and transmits the encoded data to the terminal device 3106. The capture device 3102 includes, but is not limited to, a camera, a smartphone or tablet, a computer or a laptop computer, a video conferencing system, a PDA, a vehicle-mounted device, or a combination of any of these, or the like. For example, capture device 3102 may include source device 12 as described above. When the data includes video, the video encoder 20 included in the capture device 3102 may actually perform video encoding processing. When the data includes audio (ie, voice), the audio encoder included in the capture device 3102 may actually perform audio encoding processing. For some practical scenarios, the capture device 3102 distributes encoded video and audio data by multiplexing them together. For other practical scenarios, such as in a videoconferencing system, encoded audio data and encoded video data are not multiplexed. The capture device 3102 separately distributes the encoded audio data and the encoded video data to the terminal device 3106.

In the content delivery system 3100, the terminal device 310 receives and reproduces the encoded data. The terminal device 3106 may be a device with the ability to receive and retrieve data, such as a smartphone or tablet 3108, a computer or laptop 3110, a network video recorder (NVR)/digital video recorder (DVR) 3112, a television 3114, a set-top box (STB) 3116, a videoconferencing system 3118 , a video surveillance system 3120, a personal digital assistant (PDA) 3122, a vehicle-mounted device 3124, or a combination thereof, or the like capable of decoding the above encoded data. For example, terminal device 3106 may include target device 14 as described above. When the encoded data includes video, the video decoder 30 included in the terminal apparatus is given priority to perform video decoding. When the encoded data includes audio, an audio decoder included in the terminal device is given priority to perform audio decoding processing.

For display terminal device such as 3108 smartphone or tablet, 3110 computer or laptop, 3112 Network Video Recorder (NVR)/DVR (DVR), 3114 TV, 3122 Personal Digital Assistant (PDA), 3124 Vehicle Mounted Device, the terminal device may transmit the decoded data to its display. For a terminal device not equipped with a display, such as STB 3116, video conferencing system 3118, or video surveillance system 3120, it establishes contact with an external display 3126 to receive and display decoded data.

When each device in this system performs encoding or decoding, an image encoding device or an image decoding device may be used as shown in the above embodiments.

Fig. 14 is a diagram showing the structure of an example of a terminal device 3106. After the terminal device 3106 receives a stream from the capture device 3102, the protocol processing unit 3202 parses the transmission protocol of the stream. The protocol includes, but is not limited to, Real Time Streaming Protocol (RTSP), Hypertext Transfer Protocol (HTTP), HTTP Real Time Streaming Protocol (HLS), MPEG-DASH, Real Time Transport Protocol (RTP), real-time messaging (RTMP), or any combination thereof, etc.

After the protocol processing unit 3202 processes the stream, a stream file is generated. The file is output to the demultiplexer 3204 . A demultiplexer 3204 may separate the multiplexed data into encoded audio data and encoded video data. As described above, for some practical scenarios, such as in a videoconferencing system, encoded audio data and encoded video data are not multiplexed. In this situation, the encoded data is transmitted to the video decoder 3206 and audio decoder 3208 without using the demultiplexer 3204.

Through the demultiplexing processing, a video elementary stream (ES), audio ES, and optionally subtitles are generated. The video decoder 3206, which includes the video decoder 30 as explained in the above embodiments, decodes the ES video using the decoding method as shown in the above embodiments to generate a video frame, and supplies this data to the sync block 3212. An audio decoder 3208 decodes the audio ES to generate an audio frame and provides this data to a sync block 3212. Alternatively, the video frame may be stored in a buffer (not shown in FIG. Y) before being fed to the sync block 3212. Similarly, an audio frame may be stored in a buffer (not shown in FIG. Y) before being fed into the sync block 3212.

The sync block 3212 synchronizes the video frame and the audio frame and supplies video/audio to the video/audio display 3214. For example, the sync block 3212 synchronizes the presentation of video and audio information. The information may be encoded in a syntax using timestamps regarding the presentation of encoded audio and visual data and timestamps regarding the delivery of the data stream itself.

If subtitles are included in the stream, the subtitle decoder 3210 decodes the subtitles and synchronizes them with the video frame and audio frame and passes the video/audio/subtitles to the video/audio/subtitle display 3216.

The present invention is not limited to the above system, and either the image encoding device or the image decoding device in the above embodiments may be included in another system such as an automobile system.

Although embodiments of the invention have been primarily described in terms of video coding, it should be noted that embodiments of encoding system 10, encoder 20, and decoder 30 (and system 10, respectively) and other embodiments described herein may also be configured for processing or encoding a still image, that is, processing or encoding a single image independently of any previous or subsequent image, as in video encoding. In general, only inter prediction blocks 244 (encoder) and 344 (decoder) may be unavailable if image processing coding is limited to a single image 17. All other functions (also referred to as tools or technologies) of video encoder 20 and video decoder 30 may equally be used for still image processing, such as 204/304 residual calculation, 206 transform, 206 quantization, 210/310 inverse quantization, 212/312 (inverse) transform, 262/362 split, 254/354 intra prediction, and/or 220 loop filtering, 320, entropy encoding 270, and entropy decoding 304.

The embodiments of, for example, encoder 20 and decoder 30, and the functions described herein, for example, with reference to encoder 20 and decoder 30, may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on a computer-readable medium or transmitted over a communication medium as one or more instructions or code and executed by a hardware processor. A computer-readable medium may include a computer-readable medium that corresponds to a tangible medium, such as a data medium or a communication medium, including any medium that facilitates the transfer of a computer program from one place to another, for example, in accordance with a communication protocol. Thus, computer-readable media can typically correspond to (1) tangible computer-readable storage media that are non-volatile, or (2) communication media such as a signal or carrier wave. Storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and/or data structures for implementing the methods described in this disclosure. The computer program product may include a computer readable medium.

By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk or other magnetic storage devices, flash memory, or any other media which can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the media definition. However, it should be understood that computer-readable storage media and storage media do not include connections, carrier waves, signals, or other temporary media, but are instead directed to nonvolatile tangible storage media. The disc and disc used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where discs usually reproduce data magnetically and discs reproduce data optically. with the help of lasers. Combinations of the above should also be included within the scope of computer readable media.

The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, as used herein, the term "processor" may refer to any of the aforementioned structures, or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding or included in a combined codec. In addition, the methods may be fully implemented in one or more circuits or logic elements.

The methods of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless telephone, an integrated circuit (IC), or an IC array (eg, a chipset). This disclosure describes various components, modules, or blocks to emphasize the functional aspects of devices configured to perform the disclosed technologies, but not necessarily requiring implementation in different hardware blocks. Rather, as described above, various blocks may be combined into a codec hardware block, or provided by a set of cooperating hardware blocks, including one or more processors, as described above, in combination with suitable software and/or firmware.

Claims (46)

1. A method for internal prediction of a block chromaticity sample by applying a cross-component linear model, comprising: obtaining recovered brightness samples; determining maximum and minimum luminance sample values based on the reconstructed luminance samples; obtaining the difference between the maximum and minimum values of the brightness samples; determining the position of the most significant bit of the difference between the maximum and minimum values of the brightness samples; extracting a value from a look-up table (LUT) using a set of bits as an index, the set of bits following the position of the most significant bit of the difference between the maximum and minimum values of the brightness samples; obtaining parameters α and β of the linear model based on the extracted value; and calculating the predicted value of the chrominance sample using the obtained parameters α and β of the linear model. 2. The method of claim. 1, in which the position of the most significant bit of the difference between the maximum and minimum values of the brightness samples is obtained as a base two logarithm of said difference. 3. The method according to claim 1 or 2, wherein determining the set of bits following the position of the most significant bit of the difference results in a set of bits containing four bits. 4. The method according to any one of paragraphs. 1-3, in which the most significant bit is the first non-zero bit. 5. The method according to any one of paragraphs. 1-4, wherein the method comprises: obtaining the parameters α and β of the linear model based on the extracted value and the difference between the maximum and minimum values of the restored chrominance samples. 6. The method according to p. 5, while the method contains: obtaining the parameters α and β of the linear model by multiplying the extracted value by the difference between the maximum and minimum values of the reconstructed chrominance samples. 7. The method according to any one of paragraphs. 1-6, in which the LUT contains at least three values: a first value, a second value, and a third value; among the three values, the first value and the second value are two adjacent values, and the second value and the third value are two adjacent values. 8. The method of claim 7, wherein the step between the first value and the second value is equal to the step between the second value and the third value. 9. The method of claim 7, wherein the step between the first value and the second value is different from the step between the second value and the third value. 10. The method according to any one of paragraphs. 7-9, in which the first value is indexed by the first set of bits and the second value is indexed by the second set of bits; when the value of the first set of bits is greater than the value of the second set of bits, the first value is less than the second value; or when the value of the first bit set is less than the value of the second bit set, the first value is greater than the second value. 11. The method according to any one of paragraphs. 1-10, in which the LUT contains subranges of values, and the step of any two adjacent values is constant within one subrange. 12. A device for internal prediction of a block chrominance sample by applying a cross-component linear model, wherein the device is an encoder or decoder, and the device comprises: an acquisition unit configured to obtain the reconstructed luminance samples; a determination unit configured to determine maximum and minimum values of luminance samples based on the reconstructed luminance samples; an acquisition unit, further configured to obtain a difference between the maximum and minimum values of the brightness samples; a determining unit, further configured to determine the position of the most significant bit of the difference between the maximum and minimum values of the brightness samples; and a calculation unit configured to extract a value from a look-up table (LUT) using a set of bits as an index, the set of bits following the position of the most significant bit of the difference between the maximum and minimum values of the brightness samples, obtaining linear model parameters α and β based on the extracted value; and calculating a predicted chrominance sample value using the obtained linear model parameters α and β. 13. The apparatus of claim 12, wherein the position of the most significant bit of the difference between the maximum and minimum luminance sample values is the base two logarithm of said difference. 14. The apparatus of claim 12 or 13, wherein the bit set contains four bits. 15. The device according to any one of paragraphs. 12-14, in which the most significant bit is the first non-zero bit. 16. The device according to any one of paragraphs. 12-15, wherein the calculator is configured to: obtain linear model parameters α and β based on the extracted value and the difference between the maximum and minimum values of the reconstructed chrominance samples. 17. The device according to claim 16, in which the calculation unit is configured to: obtain the parameters α and β of the linear model by multiplying the extracted value by the difference between the maximum and minimum values of the reconstructed chrominance samples. 18. The device according to any one of paragraphs. 12-17, wherein the LUT contains sub-ranges of values, and the step of any two adjacent values is constant in one sub-range. 19. An encoder containing a processing circuit for performing the method according to any one of paragraphs. 1-11. 20. A decoder containing a processing circuit for performing the method according to any one of paragraphs. 1-11. 21. A computer-readable storage medium containing program code for performing the method according to any one of paragraphs. 1-11. 22. Decoder, containing: one or more processors; and a non-volatile computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the program, when executed by the processors, configures a decoder to perform the method of any one of claims. 1-11. 23. An encoder containing: one or more processors; and a non-volatile computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the program, when executed by the processors, configures the encoder to perform the method of any one of claims. 1-11. 24. A non-volatile recording medium that includes a predictively encoded bitstream decoded by a device, the bitstream generated according to any one of paragraphs. 1-11.

Images