EA046317B1 - INTERNAL COLOR PREDICTION IN VIDEO CODING AND DECODING - Google Patents

Description

Field of technology to which the invention relates

The present invention relates to video encoding and decoding.

Prior Art

Digital video may be encoded for transmission and/or storage. This encoding has efficiency advantages in terms of channel bandwidth demand, memory requirements, and error detection and/or correction.

Video compression techniques involve taking blocks of video data and using different approaches to take advantage of redundancy. For example, intraprediction is a spatial approach that involves encoding a video block with reference to another (predictive) video block in the same frame of video data. Intra-prediction, on the other hand, is a timing approach that involves encoding a video block of a video frame with reference to a corresponding (prediction) block of another frame of video data.

Typically, video data includes luma data and chrominance data. Luminance data represents the brightness of the image without reference to color. Essentially, luminance data describes a black-and-white image, and in fact such an image can only be reconstructed from luminance data. Chroma data represents color information that can be used to restore color to a black-and-white image.

Various approaches can be used to encode luma and chrominance data. The human eye is much more sensitive to brightness than to chrominance; this makes it possible to compress chrominance data more than luma data.

List of drawing figures

In fig. 1 is a schematic diagram of a communication network in accordance with an embodiment of the invention;

in fig. 2 shows a schematic representation of the communication network emitter from FIG. 1;

in fig. 3 is a diagram illustrating an encoder implemented on the emitter of FIG. 2;

in fig. 4 is a flow diagram of the structure and functions of the encoder prediction module of FIG. 3;

in fig. 5 is a schematic representation of the communication network receiver of FIG. 1;

in fig. 6 is a diagram illustrating a decoder implemented on the receiver of FIG. 4; And;

in fig. 7 is a flow diagram of the structure and functions of the decoder prediction module of FIG. 6.

Information confirming the possibility of implementing the invention

Universal Video Coding (UVC) (MPEG-I Part 3) is a video compression standard currently being developed by the Joint Video Experts Team (JVET), a joint group of ISO/IEC JTC 1 Working Group MPEG Video Experts and the VCEG Working Group ITU-T.

Intra-prediction is typically used to exploit spatial redundancy within a frame by computing predictions based on already reconstructed samples in the immediate vicinity of the current block. The latest VHF (hereinafter referred to as VHF) design allows the use of a large number of possible internal prediction modes (hereinafter internal modes) for predicting the brightness component, including up to 65 angular (directional) modes, planar prediction and DC prediction, as well as other advanced methods forecasting.

This large number of options is achieved at the expense of a significant amount of signaling data to indicate which of the internal modes was used in the luminance data block. Conversely, to limit the influence of signaling internal modes, a reduced number of internal modes is used to internally predict the chrominance components. This includes a subset of modes available for brightness prediction, including, for example, Derived Mode (DM, corresponding to the use of the mode used for brightness prediction), as well as a small set of fixed candidates, including planar, DC, pure horizontal, and pure vertical modes.

In addition, VHF determines the use of internal modes of the Intercomponent Linear Model (ICLM). When using MCLM, the chrominance component is predicted from already reconstructed luminance samples using a linear model. Model parameters are derived using simple linear regression applied to the reconstructed neighboring luma and chrominance samples extracted from the upper-left boundaries of the encoding block. Currently, VHF uses three MCLM modes, including a general LM mode that uses the available samples in the upper left corner, and two directional LM modes that use extended arrays of reference samples extracted exclusively from above or to the left of the current block, respectively.

The use of LM prediction has been found to be effective in improving the performance of internal color prediction. However, the embodiments of the invention described herein are intended to provide additional benefits from other approaches.

- 1 046317

It is noted here that the use of simple linear forecasts can be limiting. Embodiments of the invention described herein can provide improved performance compared to existing technologies through the use of more complex architectures based on Machine Learning (ML) mechanisms.

The Hybrid Neural Network for Intra-Chroma Prediction is described in Hybrid Neural Network for Intra-Chroma Prediction (Li, Yue, Li Li, Zhu Li, Jianchao Yang, Ning Xu, Dong Liu, and Houqiang Li; 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 1797-1801. IEEE, 2018)], in which a convolutional neural network (CNN) for extracting features from reconstructed luminance samples of the current block was combined with existing fully connected architectures to extract inter-component correlations between neighboring luminance and chrominance samples. Using such an architecture, complex nonlinear mappings can be obtained for end-to-end prediction of Cb and Cr channels. However, such an architecture typically ignores the spatial correlation of adjacent samples when predicting corresponding locations in a predicted block.

Embodiments of the invention described herein provide a neural network architecture that includes an attention module for controlling the contribution of each reference neighbor sample in calculating a prediction for the location of each sample, ensuring that spatial information is taken into account when calculating the prediction. Similar to the aforementioned hybrid neural network, the method according to the embodiment disclosed herein uses a design based on three branches of the network that are combined to create prediction samples. The first two branches work simultaneously to extract features from the available reconstructed samples, including the already reconstructed luma block as well as the adjacent luma and chrominance reference samples. The first branch (called the intercomponent boundary branch) is designed to extract intercomponent information from neighboring reconstructed samples using an extended reference array to the left of and above the current block. The second branch (called the luminance convolutional branch) extracts spatial patterns over the aligned reconstructed luminance block by applying convolutional operations. The embodiments described herein present a process for combining features output by the first and second branches using an attention module. Examples of the attention module will be described below. Finally, the output of the attention module is fed to the third branch of the network to obtain the resulting output chrominance predictions Cb and Cr.

This approach, where neural networks are used to implement chroma prediction, can be considered as an extension of the existing chroma prediction modes defined in VHF. Thus, in an embodiment of the invention, a neural network based architecture can be implemented to compete with established modes for supported settings of 4x4, 8x8, 16x16. Then, for each prediction unit, the encoder chooses between traditional angular modes, LM models, or the disclosed neural network mode; by minimizing the speed distortion cost criterion.

The above arrangement may be implemented in a video communication network designed to process a video presentation using encoding techniques that allow it to be transmitted (or stored) for decoding by a playback device.

In general terms, an aspect of the present invention provides internal prediction of chroma samples based on luminance samples and an attention mask, wherein the attention mask is adjusted by a convolutional neural network.

As illustrated in FIG. 1, the apparatus is illustrated to include a schematic video communications network 10 in which emitter 20 and receiver 30 communicate via communications link 40. In practice, communications link 40 may include a satellite link, a cable network, a terrestrial broadcast network, or a POTS link. , such as those used to provide Internet service to residential and small business premises, fiber-optic communications systems, or a combination of any of the above and any other communications media.

In addition, the invention also extends to transmitting, by physical transmission, a storage medium on which a machine-readable record of an encoded bitstream is stored for passage to a suitably configured receiver capable of reading the medium and deriving the bitstream from it. An example of this is the provision of a Digital Versatile Disc (DVD) or equivalent. The following description focuses on signal transmission, for example, using an electronic or electromagnetic signal carrier, but should not be construed as excluding the above-mentioned approach including storage media.

As shown in FIG. 2, the emitter 20 is a computer device in structure and function. It may have some features in common with a general purpose computing device, but some features may be implementation specific given the specialized function for which the emitter 20 is to be used. The reader will appreciate which functions may be general purpose specific and which may require configuration, special

- 2 046317 for use in a video emitter.

Thus, the emitter 20 includes a graphics processor (GPU) 202 configured for specific use in graphics processing and the like. The emitter 20 also contains one or more other processors 204, either generally provisioned or configured for other purposes, such as mathematical operations, audio processing, communications link control, and so on.

Input interface 206 provides a means for receiving user input. Such user actions may, for example, be caused by user interaction with a particular input unit containing one or more control buttons and/or switches, a keyboard, mouse or other pointing device, a speech recognition unit configured to receive and process speech into control commands, a signal processor configured to receive and control processes from another device, such as a tablet or smartphone, or a remote control receiver. This list is intended to be non-exhaustive and the reader is encouraged to envisage other forms of input, whether user-initiated or automated.

Likewise, output interface 214 is configured to provide a means for outputting signals to a user or other device. Such a pin may include a display signal for driving a local video display unit (VDU) or any other device.

Data interface 208 implements a communication channel, whether broadcast or end-to-end, with one or more signal recipients. In the context of the present embodiment, the data interface is configured to cause emission of a signal carrying a bitstream defining a video signal encoded by the emitter 20.

The processors 204, and in particular for the benefit of the present invention, the GPU 202, are configured to execute computer programs while the encoder is operating. This utilizes the data storage facilities provided by mass storage device 208, which is configured to provide large-scale data storage, albeit on a relatively slow access basis, and in practice will store computer programs and, in the current context, video presentation data in preparation for execution. coding process.

Read Only Memory (ROM) 210 is pre-configured with executable programs designed to provide the core functionality of emitter 20, and Random Access Memory (RAM) 212 is provided for quickly accessing and storing data and program instructions during execution of a computer program.

Next, the function of the emitter 20 will be described with reference to FIG. 3. In FIG. 3 shows the processing performed by the encoder implemented on the emitter 20 using executable commands on a data file that is a video presentation containing a plurality of frames for sequential display as a sequence of images.

The data file may also contain audio playback information accompanying the video presentation and other additional information, such as electronic program manual information, subtitles, or metadata to enable cataloging of the presentation. The handling of these aspects of the data file is not relevant to the present invention.

As shown in FIG. 3, the current image or frame in a sequence of images is sent to the division module 230, where it is divided into rectangular blocks of a given size for processing by the encoder. This processing can be serial or parallel. The approach may depend on the processing capabilities of the particular implementation.

Each block is then input to prediction module 232, which seeks to eliminate the temporal and spatial redundancy of data present in the sequence and obtain a prediction signal using the previously encoded content. The information that allows such a prediction to be computed is encoded in the bitstream. This information must be sufficient to perform calculations, including the ability to extract, through analysis at the receiver, other information necessary to complete the prediction. Further details of the prediction module of this embodiment of the invention will be provided below.

The prediction signal is subtracted from the original signal to obtain the residual signal. It is then input to transform module 234, which attempts to further reduce the spatial redundancy of data within the block by using a more appropriate data representation. As noted above, this embodiment assumes that a domain transformation cannot be implemented for every set of residues, and depending on the likely efficiency of doing so, skipping the transformation may be implemented instead. The use of skip conversion may be signaled in the bitstream.

The resulting signal is then typically quantized by a quantization module 236, and finally the resulting data, formed from the coefficients and information needed to calculate the prediction for the current block, is input to an entropy encoding module 238, which uses statistical redundancy to represent the signal in a compact form using short

- 3 046317 binary codes. The reader will note that if transform skipping was used, transform skip residual coding (TSRC) is used.

Upon repeated exposure to the emitter encoding means 20, a bit stream of block information elements can be constructed for transmission to a receiver or multiple receivers, as appropriate. The bitstream may also contain information elements that apply to multiple block information elements and are thus held in the bitstream syntax independent of the block information elements. Examples of such information elements include configuration options, parameters applicable to a sequence of frames, and parameters related to the video presentation as a whole.

Next, the prediction module 232 will be described in more detail with reference to FIG. 4. As will be appreciated, this is just an example and other approaches may be contemplated within the scope of the present invention and the appended claims.

The following process is executed on each block in the frame.

Prediction module 232 is configured to process the input luma samples and chrominance samples to produce prediction data, which can then be used to generate residuals for further processing as described above.

We will now describe the structure and functions of the prediction module 232 with reference to FIG. 4.

In general, prediction module 232 comprises a neural network architecture that includes attention module 250 for controlling the contribution of each reference neighbor sample in calculating a prediction for each sample location, allowing spatial information to be taken into account in calculating the prediction. Prediction module 232 has three network branches that combine to create prediction samples. As shown, the first and second branches are implemented in parallel to extract features from the available samples, including the luma block as well as the adjacent luma and chrominance reference samples. The first branch (called the inter-component boundary branch) is designed to extract inter-component information from neighboring reconstructed samples using an extended reference array to the left of and above the current block. The second branch (called the luminance convolutional branch) extracts spatial patterns over the aligned luminance block by applying convolutional operations. The attention module combines the functions output by the first and second branches.

The output of the attention module 250 is finally fed to the third branch of the network to obtain the resulting output chrominance predictions Cb and Cr.

Attention-based learning used within deep learning is used to improve the performance of trained networks on complex prediction tasks. Attention models can reduce task complexity by predicting smaller areas of attention that are processed sequentially to promote more efficient learning. Specifically, self-attention (or internal attention) is used to evaluate the influence of certain input variables on outputs, with a prediction computed focusing on the most relevant elements of the same sequence. The attention module 250 of the described embodiment attempts to combine features from the first and second branches of the network to estimate the influence of each input variable with respect to their spatial location.

This approach can address an important limitation of similar methods for computing intrapredictive ML-based samples, which typically completely discard the information contained in the spatial location of input and output samples.

In an example use of an embodiment of the invention, it is assumed that intraprediction has already been performed on the luminance component and that the final reconstructed luminance samples are available. Depending on the chroma subsample used, this block is then downsampled using conventional filters available in VHF to produce a block of fused luminance samples with the same NxN dimensions in space as the corresponding chroma components. For simplicity, this version of the algorithm considers only square blocks. Larger block sizes, as well as advanced downsampling filters, can be used without loss of generality.

The combined luminance block is denoted X e R ^NxN . The array of reference samples in the upper left corner of the current block is denoted by B _c e R ^2N+1 , b = 2N+1, where c = Y, Cb, Cr for the three components respectively, where B is constructed as follows.

First, the samples on the left boundary are considered starting with the lowest sample; then the angle is calculated; then the samples from the top are considered, starting with the leftmost sample. If some reference samples are not available, they are padded with a predetermined value.

Finally, the intercomponent volume obtained by concatenating the three reference arrays B _y , B _Cb , B _Cr is denoted S e R ^3xb . After applying the intercomponent boundary branch over S and the convolutional brightness branch over X, the transformed feature volumes X ε ;κ ^{ΝχΝχΰ=Ν2χ} <- and § e rR ^bxD are obtained, respectively.

- 4 046317

Figure 4 illustrates the subnet architecture for both branches (for the special case of an 8x8 block). The table shows the disclosure of selected network hyperparameters for other block sizes. Details of the construction and implementation of both branches are not given due to the use of a structure similar to the above-mentioned modern hybrid method based on a neural network.

Table - network hyperparameters depending on block size. In particular, the depth of convolutional kernels is applied to different branches of the proposed architecture.______

Branch 4x4 8x8 16 x 16 Intercomponent boundary branch 16, 32 32, 64 64, 96 Convolutional brightness branch 32, 32 64, 64 96, 96 Attention module 16, 16, 32 16, 16, 64 16, 16, 96 Output branch 32.2 64.2 96.2

Already within the proposed attention-based fusion module, each of these two feature maps is convolved using a 1x1 kernel to project them onto two corresponding reduced feature spaces. Specifically, S is convolved with the filter W _F e Rhx ^D to obtain an h-dimensional feature matrix F. Similarly, X is convolved with the filter WG x R ^h x ^C to obtain an h-dimensional feature matrix G. The two matrices are multiplied to obtain a preliminary map attention M=G ^T F. Finally, the attention matrix AeA ^N2xb is obtained by applying the multivariable logistic function operation to each element of M to product the probability that each edge location can predict the location of each sample in the block. Formally, for each element ai,j in A, where j = 0...N2-1 represents the sample location in the forecast block, aj = 0...b - 1$ represents the reference sample location, the following applies:

mj,j е Т where T is the temperature parameter of the multivariable logistic function, which controls the smoothness of the generated probabilities, for 0<T<1. Note that the smaller the value of T, the more localized the resulting attention regions are, resulting in a correspondingly smaller number of edge samples contributing to a given predicted location.

The weighted sum of the contribution of each reference sample to predicting a given output location is obtained by calculating the dot product of the intercomponent boundary features S and the attention matrix A, or formally &A>, where O is the dot product. To further refine <§, A) this weighted sum can be multiplied by the output of the brightness branch. To do this, the luma branch output must be transformed to resize it through a 1x1 convolution using the ^w x matrix to obtain the transformed representation X eT? ^N2 * ^D . This operation is used to ensure that the output sizes of the luminance convolutional branch are compatible with <YA). Finally, the output of the attention model is 0 e/? ^Ν2χΕ) is obtained as follows: X ^o (§ _? A>, where ^o is the element-wise product.

For example, the dimension parameter h in the attention module is set to 16 for all trained models, which is an optimal trade-off between performance and complexity. The value of T = 0.5 is cross-sectional, providing an optimal balance between samples that are informative and those that are noisy from the remaining edge locations.

Additionally, it should be noted that prediction module 232 may be configured to use one of a variety of predefined inter-prediction modes, but inter-prediction is not within the scope of the present invention. Moreover, the above invention does not exclude the possibility of implementing existing internal forecasting modes specified, for example, in VHF.

The structural architecture of the receiver is shown in FIG. 5. It has elements that represent a computer device. Thus, receiver 30 includes a graphics processor 302 configured for specific use in graphics processing and the like. Receiver 30 also contains one or more other processors 304, either generally provisioned or configured for other purposes, such as mathematical operations, audio processing, communications link control, and so on.

As will be appreciated by the reader, receiver 30 may be implemented as a set-top box, portable personal electronic device, personal computer, or any other device suitable for playing video presentations.

Input interface 306 provides means for receiving user input. Such user input actions may, for example, be caused by the user interacting with a particular input block containing one or more control buttons and/or switches, a keyboard, mouse or other pointing device, a speech recognition block configured with

- 5 046317 the ability to receive and process speech into control commands, a signal processor configured to receive and control processes from another device, such as a tablet or smartphone, or a remote control receiver. This list is intended to be non-exhaustive and other forms of input, whether user initiated or automated, may be provided.

Likewise, output interface 314 is configured to provide a means for outputting signals to a user or other device. Such output may include a television signal in a suitable format for driving a local television device.

Data interface 308 implements a communication channel, whether broadcast or end-to-end, with one or more signal recipients. In the context of the present embodiment, the data interface is configured to cause emission of a signal carrying a bitstream defining a video signal to be encoded by the receiver 30.

The processors 304, and in particular for the benefit of the present invention, the GPU 302, are configured to execute computer programs while the receiver is operating. This utilizes the data storage facilities provided by mass storage device 308, which is configured to provide large-scale data storage, albeit on a relatively slow access basis, and in practice will store computer programs and, in the current context, video presentation data resulting from completing the admission process.

Read Only Memory (ROM) 310 is preconfigured with executable programs to provide the core functionality of receiver 30, and Random Access Memory 312 is provided for quickly accessing and storing data and program instructions during execution of a computer program.

Next, the function of the receiver 30 will be described with reference to FIG. 6. In FIG. 6 shows the processing performed by the decoder implemented on the receiver 20 using executable instructions on a bit stream received at the receiver 30 containing structured information from which a video presentation can be derived containing a reconstruction of the frames encoded by the encoder functionality of the emitter 20.

The decoding process shown in FIG. 6 aims to reverse the process performed on the encoder. The reader will appreciate that this does not mean that the decoding process is strictly the reverse of the encoding process.

The received bit stream contains a sequence of encoded information elements, with each element associated with a block. The block information element is decoded in entropy decoding module 330 to obtain a block of coefficients and information necessary to compute a prediction for the current block. The block of coefficients is typically dequantized in dequantization module 332 and typically converted back to the spatial domain by transform module 334, unless the decoder is notified to skip conversion.

As noted above, the reader will understand that entropy decoding, dequantization, and inverse conversion should only be used at the receiver if entropy encoding, quantization, and conversion, respectively, were used at the emitter.

The prediction signal is generated, as before, from previously decoded samples from current or previous frames and using information decoded from the bitstream by prediction module 336. A reconstruction of the original image block is then obtained from the decoded residual signal and the calculated prediction block in reconstruction block 338. In embodiments, prediction module 336 responds to bitstream information signaling the use of intra prediction and, if such information is present, to readings from that bitstream information that allow the decoder to determine which intra prediction mode was used and thus , which prediction method should be used when reconstructing a block data sample.

By repeatedly exposing sequentially received block information elements to the decoding functionality, the graphic blocks can be reconstructed into frames, which can then be assembled to create a video presentation for playback.

An example of a decoder algorithm complementary to the encoder algorithm described previously is shown in FIG. 7. This algorithm refers to the use of a chroma prediction mode implemented by a neural network, if the implementation also allows other chroma prediction modes to be implemented as such modes are currently defined in VHF, then this can also be communicated to the decoder.

As noted previously, the decoding functionality of the receiver 30 extracts from the bitstream the sequence of block information elements encoded by the emitter encoder 20, the block defining information, and associated configuration information.

The following process is performed for each encoded block in the frame to be decoded.

The prediction module 332 is configured to process the input encoded data

-

Claims (10)

luminance and encoded chrominance data to obtain reconstructed sample luminance and chrominance data, which can then be used for further processing as described above. We will now describe the structure and function of the prediction module 332 with reference to FIG. 7. In general, prediction module 332 comprises a neural network architecture that includes attention module 350 for controlling the contribution of each reference neighbor sample in calculating a prediction for each sample location, allowing spatial information to be taken into account in calculating the prediction. Prediction module 332 has three network branches that combine to create prediction samples. As shown, the first and second branches are implemented in parallel to extract features from the available reconstructed samples, including the already reconstructed luma block, as well as the adjacent luma and chrominance reference samples. The first branch (called the inter-component boundary branch) is designed to extract inter-component information from neighboring reconstructed samples using an extended reference array to the left of and above the current block. The second branch (called the luminance convolutional branch) extracts spatial patterns over the aligned luminance block by applying convolutional operations. The attention module combines the functions output by the first and second branches. The output of the attention module 350 is finally fed to the third branch of the network to obtain the resulting Cb and Cr chrominance prediction outputs. The decoder uses the same approach to the operation of the attention module as the encoder. The attention module in the decoder is expected to be trained in the same way as in the encoder to reproduce the internal prediction decisions made in the encoder. This ensures that the chroma data is restored in the decoder. It should be understood that the invention is not limited to the embodiments described above, and various modifications and improvements can be made without departing from the concepts described. Unless mutually exclusive, any of the features may be used alone or in combination with any other features, and the present invention extends to include all combinations and subcombinations of one or more of the features described herein. CLAIM 1. A method for decoding video data, which includes: extracting reference samples from the reconstructed luma samples and the reconstructed chroma samples; and constructing at least one block of chroma prediction samples from the reference samples, wherein the construction of the chroma prediction samples depends on the spatial arrangement of the reference samples, and wherein the construction of the chroma prediction samples depends on the use of the attention module. 2. The method of claim 1, wherein the reference samples include reconstructed chroma samples from adjacent blocks of said block of chroma prediction samples. 3. The method of claim 1 or 2, wherein the reference samples include reconstructed luma samples adjacent to the block of chroma prediction samples. 4. The method of claim 1, 2, or 3, wherein the reference samples include reconstructed luminance samples from adjacent blocks of a block of adjacent reconstructed luminance samples. 5. The method as claimed in any one of the preceding claims, wherein the attention module is configured as a deep neural network. 6. The method as claimed in any one of the preceding claims, wherein the construction of color prediction samples depends on the calculation of intercomponent information. 7. The method as claimed in any one of the preceding claims, wherein the construction of the chrominance prediction samples depends on retrieving spatial image data over a block of luminance data using at least one convolutional operation. 8. The method as claimed in any one of the preceding claims, including controlling, by the attention module, the contribution of each reference neighbor sample to a prediction calculation for the sample location. 9. The method of any one of the preceding claims, further comprising one or more other construction modes of the at least one chrominance data block, and wherein the construction mode of the at least one chrominance data block is determined based on the received signal. 10. A decoder for decoding video data, including: a reference sample extractor for extracting reference samples from the reconstructed luma samples and the reconstructed chrominance samples; and a chroma prediction sample constructor for constructing at least one block of chroma prediction samples from the reference samples, where the prediction sample constructor -