TW202243476A - A front-end architecture for neural network based video coding - Google Patents

Abstract

Techniques are described herein for processing video data using a neural network system. For instance, a process can include generating, by a first convolutional layer of an encoder sub-network of the neural network system, output values associated with a luminance channel of a frame. The process can include generating, by a second convolutional layer of the encoder sub-network, output values associated with at least one chrominance channel of the frame. The process can include generating, by a third convolutional layer based on the output values associated with the luminance channel of the frame and the output values associated with the at least one chrominance channel of the frame, a combined representation of the frame. The process can further include generating encoded video data based on the combined representation of the frame.

Description

Front-end architecture for neural network-based video decoding

In a nutshell, this case concerns image and video coding, including the encoding (or compression) and decoding (decompression) of images and/or video. For example, aspects of this case relate to image and video coding systems for processing luma-chrominance (YUV) input formats (e.g., 4:2 :0 YUV input format, 4:4:4 YUV input format, 4:2:2 YUV input format, etc.) and/or other input format technologies.

Many devices and systems allow video data to be processed and output for consumption. Digital video data includes a large amount of data to meet the needs of consumers and video providers. For example, consumers of video data expect high-quality video, including high fidelity, high resolution, high frame rate, and the like. As a result, the large amount of video data required to meet these demands places a burden on the communication networks and equipment that process and store the video data.

Various video decoding techniques can be used to compress video data. One goal of video decoding is to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation to video quality. As ever-evolving video services become available, coding techniques with better coding efficiency are required.

Systems and techniques are described for decoding (eg, encoding and/or decoding) imagery and/or video content using one or more machine learning systems. For example, provides an end-to-end machine learning (e.g., neural network) based image and video coding (E2E-NNVC) system that can handle the YUV (digital domain YCbCr) input format (and in some cases, other input format), in some cases specifically the 4:2:0 YUV input format. The E2E-NNVC system can process individual frames (also called images or pictures) and/or video data consisting of multiple frames. The YUV format includes a luma channel (Y) and a pair of chroma channels (U and V). The U and V channels can be subsampled relative to the Y channel without noticeable or noticeable impact on visual quality. In YUV format, the correlation between channels is reduced, which may be different from other color formats such as red-green-blue (RGB) format. Aspects of the systems and techniques described herein provide front-end architectures (e.g., new subnetworks) to accommodate E2E-NNVC designed for RGB input formats (and in some cases, E2E-NNVC designed for other input formats). NNVC) in the YUV 4:2:0 input format (and in some cases, other input formats). The front-end architecture is applicable to many E2E-NNVC architectures.

In one illustrative example, a method of processing video data is provided. The method includes: generating, by a first convolutional layer of an encoder subnetwork of a neural network system, an output value associated with a luma channel of a frame; and generating, by a second convolutional layer of the encoder subnetwork, an output value associated with the frame an output value associated with at least one chroma channel of the frame; generated by the third convolutional layer based on the output value associated with the luma channel of the frame and the output value associated with the at least one chroma channel of the frame a combined representation of the frame; and generating encoded video data based on the combined representation of the frame.

In another example, an apparatus for processing video data is provided, comprising: a memory; and a processor (eg, implemented in a circuit) coupled to the memory. In some instances, more than one processor may be coupled to the memory and may be used to perform one or more of these operations. The processor is configured to: use a first convolutional layer of an encoder subnetwork of the neural network system to generate an output value associated with a luma channel of a frame; use a second convolutional layer of the encoder subnetwork to generate an output value associated with an output value associated with at least one chroma channel of the frame; using a third convolutional layer based on an output value associated with the luma channel of the frame and an output associated with the at least one chroma channel of the frame generating a composite representation of the frame; and generating encoded video data based on the composite representation of the frame.

In another example, a non-transitory computer-readable medium for encoding video data is provided, having stored thereon instructions that, when executed by one or more processors, cause the The one or more processors perform the following operations: using a first convolutional layer of an encoder subnetwork of the neural network system to generate an output value associated with a luma channel of a frame; using a second convolutional layer of the encoder subnetwork layer produces an output value associated with at least one chrominance channel of the frame; using a third convolutional layer based on the output value associated with the luma channel of the frame and the at least one chrominance channel associated with the frame generating a composite representation of the frame from output values of the frames; and generating encoded video data based on the composite representation of the frame.

In another example, an apparatus for processing video data is provided. The device comprises: a unit for generating an output value associated with a luma channel of a frame via a first convolutional layer of an encoder subnetwork of a neural network system; a unit for via a second convolutional layer of the encoder subnetwork a convolutional layer to generate an output value associated with at least one chrominance channel of the frame; for using a third convolutional layer based on the output value associated with the luma channel of the frame and the output value associated with the frame means for generating a combined representation of the frame from the associated output value of the at least one chrominance channel; and means for generating encoded video data based on the combined representation of the frame.

In some aspects, the third convolutional layer includes a 1×1 convolutional layer. The 1x1 convolution layer includes one or more 1x1 convolution filters.

In some aspects, the above method, apparatus, and computer readable medium for processing video data also include: using the first nonlinear layer of the encoder sub-network to process the luminance channel associated with the frame and processing the output values associated with the at least one chrominance channel of the frame using a second nonlinear layer of the encoder subnetwork. In such aspects, the combined representation is generated based on the output of the first nonlinear layer and the output of the second nonlinear layer.

In some aspects, the combined representation of the frame is generated by the third convolutional layer using the output of the first nonlinear layer and the output of the second nonlinear layer as input.

In some aspects, the above-mentioned method, apparatus and computer-readable medium for processing video data also include: quantizing the encoded video data.

In some aspects, the above-mentioned method, apparatus and computer-readable medium for processing video data also includes: performing entropy decoding on the encoded video data.

In some aspects, the above method, device and computer-readable medium for processing video data also include: storing the encoded video data in a memory.

In some aspects, the above-mentioned method, apparatus and computer-readable medium for processing video data also include: sending the encoded video data to at least one device on a transmission medium.

In some aspects, the above method, apparatus, and computer readable medium for processing video data also include: obtaining encoded frames; generated by the first convolutional layer of the decoder sub-network of the neural network system a reconstructed output value associated with the luma channel of the encoded frame; and a reconstructed output value associated with at least one chrominance channel of the encoded frame produced by the second convolutional layer of the decoder subnetwork Structured output value.

In some aspects, the above method, apparatus, and computer readable medium for processing video data also include: using the third convolutional layer of the decoder subnetwork to combine the luma channel of the encoded frame with the The at least one chrominance channel of the encoded frame is separated.

In some aspects, the third convolutional layer of the decoder sub-network includes a 1x1 convolutional layer. The 1x1 convolution layer includes one or more 1x1 convolution filters.

In some aspects, the frame includes a video frame. In some aspects, the at least one chroma channel includes a chroma blue channel and a chroma red channel. In some aspects, the frame has a luma-chroma (YUV) format.

In one illustrative example, a method of processing video data is provided. The method includes: obtaining an encoded frame; separating, by a first convolutional layer of the decoder subnetwork, a luma channel of the encoded frame from at least one chrominance channel of the encoded frame; The second convolutional layer of the decoder subnetwork of the network system produces the reconstructed output value associated with the luma channel of the encoded frame; the third convolutional layer of the decoder subnetwork produces the output value associated with the reconstructed output value associated with the at least one chrominance channel of the encoded frame; and generating an output frame comprising the reconstructed output value associated with the luma channel and the A reconstructed output value associated with at least one chroma channel.

In another example, an apparatus for processing video data is provided, comprising: a memory; and a processor (eg, implemented in a circuit) coupled to the memory. In some instances, more than one processor may be coupled to the memory and may be used to perform one or more of these operations. The processor is configured to: obtain an encoded frame; use the first convoluted layer of the decoder subnetwork to combine a luma channel of the encoded frame with at least one chrominance channel of the encoded frame Separation; using the second convolutional layer of the decoder subnetwork of the neural network system to generate a reconstructed output value associated with the luma channel of the encoded frame; using the third convolutional layer of the decoder subnetwork a convolutional layer produces reconstructed output values associated with the at least one chrominance channel of the encoded frame; and produces an output frame comprising the reconstructed output associated with the luma channel value and a reconstructed output value associated with the at least one chroma channel.

In another example, a non-transitory computer-readable medium for encoding video data is provided, having stored thereon instructions that, when executed by one or more processors, cause the one or more processors: obtain an encoded frame; use the first convoluted layer of the decoder sub-network to combine the luma channel of the encoded frame with at least one of the encoded frame chrominance channel separation; using a second convolutional layer of the decoder subnetwork of the neural network system to generate a reconstructed output value associated with the luma channel of the encoded frame; using the decoder subnetwork The third convolutional layer produces the reconstructed output value associated with the at least one chrominance channel of the encoded frame; and produces an output frame comprising the reconstructed output value associated with the luma channel The reconstructed output value and the reconstructed output value associated with the at least one chroma channel.

In another example, an apparatus for processing video data is provided. The apparatus comprises: a unit for obtaining an encoded frame; and at least one of a luminance channel of the encoded frame and the encoded frame via a first convoluted layer of the decoder subnetwork. a unit for chrominance channel separation; a unit for generating, via a second convolutional layer of a decoder subnetwork of a neural network system, a reconstructed output value associated with the luma channel of the encoded frame; with means for generating a reconstructed output value associated with the at least one chrominance channel of the encoded frame through a third convolutional layer of the decoder subnetwork; and means for generating an output frame, The output frame includes reconstructed output values associated with the luma channel and reconstructed output values associated with the at least one chrominance channel

In some aspects, the first convolutional layer of the decoder sub-network includes a 1x1 convolutional layer. The 1x1 convolution layer includes one or more 1x1 convolution filters.

In some aspects, the above method, apparatus and computer readable medium for processing video data also includes: using the first non-linear layer of the decoder sub-network to process the luminance channel of the encoded frame associated values, wherein the reconstructed output values associated with the luma channel are generated based on the output of the first nonlinear layer; and using the second nonlinear layer of the decoder subnetwork to processing values associated with the at least one chroma channel of the encoded frame, wherein the reconstructed output values associated with the at least one chroma channel are based on the output of the second nonlinear layer produced.

In some aspects, the above method, apparatus and computer readable medium for processing video data also includes: dequantizing the samples of the encoded frame.

In some aspects, the above method, apparatus and computer readable medium for processing video data also includes entropy decoding the encoded frame samples.

In some aspects, the above-mentioned method, device and computer-readable medium for processing video data also include: storing the output frame in a memory.

In some aspects, the above method, device and computer-readable medium for processing video data also include: displaying the output frame.

In some aspects, the above-mentioned method, apparatus, and computer readable medium for processing video data also include: generating, by the first convolutional layer of the encoder sub-network of the neural network system, the output value associated with at least one chrominance channel of the frame is generated by the second convolutional layer of the encoder subnetwork; the output value associated with at least one chrominance channel of the frame is generated by the third convolutional layer of the encoder subnetwork based on the generating a combined representation of the frame from an output value associated with the luma channel of the frame and an output value associated with the at least one chrominance channel of the frame; and generating the encoded representation based on the combined representation of the frame frame.

In some aspects, the third convolutional layer of the encoder sub-network includes a 1x1 convolutional layer. The 1x1 convolution layer includes one or more 1x1 convolution filters.

In some aspects, the above method, apparatus, and computer readable medium for processing video data also include: using the first nonlinear layer of the encoder sub-network to process the luminance channel associated with the frame an output value; and processing the output value associated with the at least one chrominance channel of the frame using a second nonlinear layer of the encoder subnetwork; wherein the combined representation is based on the output of the first nonlinear layer and the output of this second nonlinear layer is generated.

In some aspects, the combined representation of the frame is generated by the third convolutional layer of the encoder subnetwork using the output of the first nonlinear layer and the output of the second nonlinear layer as input .

In some aspects, the encoded frame includes an encoded video frame.

In some aspects, the at least one chroma channel includes a chroma blue channel and a chroma red channel.

In some aspects, the encoded frame has a luma-chroma (YUV) format.

In some aspects, the device may be, or may be part of, a mobile device (eg, a mobile phone or so-called "smart phone," tablet computer or other type of mobile device), a network Connected wearable devices, extended reality devices (for example, virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), personal computers, laptops, server computers (for example, video servers or other server devices), televisions, vehicles (or computing devices or systems for vehicles), cameras (e.g., digital cameras, Internet Protocol (IP) cameras, etc.), multi-camera systems, robotic devices or systems, aviation device or system, or other device. In some aspects, the device also includes at least one camera for capturing one or more images or video frames (or pictures). For example, the device may include a camera (eg, an RGB camera) or cameras for capturing one or more images including video frames and/or one or more videos. In some aspects, the device includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the apparatus includes a transmitter configured to send one or more video frames and/or syntax data to at least one device over a transmission medium. In some aspects, the apparatus described above may include one or more sensors. In some aspects, a processor includes a neural processing unit (NPU), central processing unit (CPU), graphics processing unit (GPU), or other processing device or component.

This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. An understanding of the subject matter should be understood by reference to the appropriate portion of the entire specification of this patent, any or all drawings and each claim.

The foregoing and other features and embodiments will become more apparent upon reference to the following specification, claims and drawings.

Certain aspects and examples of the subject matter are provided below. As will be apparent to those having ordinary knowledge in the art to which the present invention pertains, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present case. It will be apparent, however, that various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting.

The ensuing description provides example embodiments only, and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the ensuing description of the example embodiments will provide those having ordinary skill in the art to which the invention pertains with an enableable description for implementing the example embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the disclosure as set forth in the appended claims.

Digital video data can include large amounts of data, especially as the demand for high-quality video data continues to grow. For example, consumers of video content typically expect increasingly higher quality video with high fidelity, high resolution, high frame rate, and the like. However, the large amount of video data required to meet such demands can place a significant burden on communication networks and devices that process and store the video data.

Various techniques may be used to decode the video data. Video coding may be performed according to a specific video coding standard. Example video coding standards include High Efficiency Video Coding (HEVC), Improved Video Coding (AVC), Motion Picture Experts Group (MPEG) Coding, Versatile Video Coding (VVC), and others. Video decoding typically uses predictive methods such as inter-frame prediction or intra-frame prediction, which exploit the redundancy present in video images or sequences. A common goal of video decoding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing video quality degradation. As the demand for video services grows and new video services become available, coding techniques with better coding efficiency, performance, and rate control are required.

Described herein are systems, apparatus, programs (also referred to as methods), and computer-readable media (collectively "systems and technology"). In general, ML is a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and reasoning without the use of explicit instructions. One example of a ML system is a neural network (also known as an artificial neural network), which may include a set of interconnected artificial neurons (eg, a neuronal model). Neural networks can be used in various applications and/or devices, such as image and/or video decoding, image analysis and/or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles , service robots, etc.

Individual nodes in a neural network can be compared to biological neurons by taking input data and performing simple operations on the data. The results of simple operations performed on input data are selectively passed to other neurons. Weight values are associated with each vector and node of the network, and these values constrain how the input data is related to the output data. For example, the input data for each node can be multiplied by the corresponding weight value, and the products can be summed. The sum of the products can be adjusted via an optional bias, and an activation function can be applied to the result, producing the node's output signal or "output activation" (sometimes called an activation map or feature map). The weight values may initially be determined by an iterative flow of training data through the network (eg, weight values are established during a training phase in which the network learns how to recognize a particular class via its typical input data characteristics).

There are different types of neural networks such as convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), multilayer perceptron (MLP) neural networks, etc. For example, a Convolutional Neural Network (CNN) is a type of feed-forward artificial neural network. A convolutional neural network may include a collection of artificial neurons, each with a receptive field (eg, a spatially localized region of the input space) that collectively tiles the input space. RNNs work by saving the output of a layer and feeding that output back into the input to help predict the outcome of the layer. A GAN is a form of generative neural network that learns patterns in input data such that the neural network model can produce new synthetic outputs that might reasonably have come from the original data set. GANs can include two neural networks operating together, including a generative neural network that produces a synthetic output and a discriminative neural network that evaluates the authenticity of the output. In an MLP neural network, data can be fed to an input layer, and one or more hidden layers provide a level of abstraction for the data. Subsequently, predictions can be made on the output layer based on the abstracted material.

In a layered neural network architecture (when there are multiple hidden layers, it is called a deep neural network), the output of the first layer of artificial neurons becomes the input of the second layer of artificial neurons, and the second layer of artificial neurons The output of the neuron becomes the input of the artificial neurons of the third layer, and so on. For example, a CNN can be trained to recognize hierarchies of features. Computation in a CNN architecture can be distributed over a population of processing nodes that can be configured in one or more computational chains. These multilayer architectures can be trained one layer at a time and can be fine-tuned using backpropagation.

In some aspects, the systems and techniques described herein include end-to-end ML-based (e.g., using a neural network architecture) image and video designed to process input data having a luminance-chrominance (YUV) input format Decoding (E2E-NNVC) system. The YUV format includes a luma channel (Y) and a pair of chroma channels (U and V). The U channel may be referred to as the chroma (or chroma)-blue channel, and the V channel may be referred to as the chroma (or chroma)-red channel. In some cases, the luma (Y) channel or component may also be referred to as a luma channel or component. In some cases, the chroma (U and V) channels or components may also be referred to as chroma channels or components. YUV input formats may include YUV 4:2:0, YUV 4:4:4, YUV 4:2:2, etc. In some cases, the systems and techniques described herein can be designed to handle other input formats, such as with Y-chroma blue (Cb)-chroma red (Cr) (YCbCr) format, red-green-blue (RGB ) format and/or other formats. The E2E-NNVC system described herein can encode and/or decode individual frames (also called images or pictures) and/or video data comprising multiple frames.

In many cases, an E2E-NNVC system is designed as an autoencoder subnetwork (encoder subnetwork) responsible for learning a probabilistic model of quantized latent variables (latent) for entropy decoding (decoder subnetwork). road) and the combination of the second subnetwork (in some cases, also known as the super prior network). In some cases, other subnetworks of decoders may exist. This type of E2E-NNVC system architecture can be regarded as a combination of transform plus quantization module (or encoder subnetwork) and entropy modeling subnetwork module.

Most E2E-NNVC system architectures are designed to operate with non-subsampled input formats, such as RGB, YUV 4:4:4 or other non-subsampled input formats. However, video coding standards such as HEVC and VVC are designed to support the YUV 4:2:0 color format in their respective main profiles. To support the 4:2:0 YUV format, the E2E-NNVC architecture, which is designed to operate with non-subsampled input formats, must be modified.

The systems and techniques described herein provide a front-end architecture (eg, sub-network) for processing one or more specific color formats (eg, YUV 4:2:0 color format) applicable to existing E2E-NNVC architectures. The systems and techniques take into account the different characteristics and differences in resolution of the Y and UV channels. For example, the Y and UV channels of a frame or a portion of a frame can be input to two separate neural network layers of the encoder sub-network of the neural network system. In some examples, the two neural network layers include convolutional layers. In some aspects, the outputs of the two separate neural network layers are processed by a pair of non-linear layers or operators of the encoder sub-network. The pair of nonlinear layers or operands may include a generalized divisor normalization (GDN) layer or operand, a parameter rectified linear unit (PReLU) layer or operand, and/or other nonlinear layers or operands. Use an additional neural network layer of the encoder subnetwork to combine the outputs of two separate neural network layers (or outputs of nonlinear layers or operators).

In some examples, the additional neural network layer is a 1x1 convolutional layer. The 1x1 convoluted layer performs per-pixel or per-value cross-channel mixing of Y and UV components (eg, via generating a linear combination), resulting in cross-component (eg, cross-luminance and chrominance) prediction that improves coding performance. For example, cross-channel mixing of Y and UV components decorrelates the Y component from the U and V components, which results in improved coding performance (eg, increased coding efficiency). In some cases, the 1x1 convolutional layer may include N 1x1 convolutional filters (where N is equal to an integer value corresponding to the number of channels input to the 1x1 convolutional layer). Each 1x1 convolution filter has a respective scaling factor applied to the corresponding nth channel of the Y component and the corresponding nth channel of the UV component.

The output of additional neural network layers (e.g., 1x1 convolutional layers) can be processed by one or more non-linear layers of the encoder subnetwork and/or one or more additional neural network layers (e.g., convolutional layers) . The quantization engine may perform quantization on the features output by the final neural network layer of the encoder sub-network to produce a quantized output. An entropy encoding engine may entropy encode the quantized output from the quantization engine to generate a bitstream. The neural network system can output a bit stream for storage, for transmission to another device, a server device or system, and the like.

The decoder subnetwork of the NNS or the decoder subnetwork of another NNS (another device) can decode the bitstream. For example, the entropy decoding engine of the decoder sub-network can entropy decode the bitstream and output the entropy decoded data to the dequantization engine. The dequantization engine can dequantize data. The dequantized data may be processed by one or more neural network layers (eg, convolutional layers) and/or one or more inverse nonlinear layers of the decoder sub-network. For example, a 1x1 convolutional layer may process data after being processed by one or more convolutional layers and one or more inverse nonlinear layers. The 1x1 convoluted layer can divide the data into Y channel features and combined UV channel features. The Y channel feature and the combined UV channel feature may be processed by two final neural network layers (eg, two convolutional layers), and in some cases by two final inverse nonlinear layers. For example, the first final neural network layer may process the Y channel features and output a reconstructed Y channel of each pixel or sample (eg, luma samples or pixels) of the reconstructed frame. The second final neural network layer can process the combined UV channel features, and can output the reconstructed U channel of each primitive or sample of the reconstructed frame (e.g., chroma blue samples or primitives) and Reconstructed V channel for each pixel or sample of the reconstructed frame (eg, chroma red samples or pixels).

Additional details regarding the systems and techniques will be described with reference to the figures.

1 illustrates an example implementation of a system on chip (SOC) 100, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with computing devices (e.g., neural networks with weights), delays, frequency band information, task information, and other information can be stored in the neural processing memory block associated with CPU 102, stored in memory block associated with graphics processing unit (GPU) 104, stored in memory block associated with graphics processing unit (GPU) 104, The associated memory block of the digital signal processor (DSP) 106 is stored in the memory block 118 and/or may be distributed among multiple blocks. Instructions executed at CPU 102 may be loaded from program memory associated with CPU 102 or may be loaded from memory block 118 .

SOC 100 may also include additional processing blocks customized for specific functions, such as GPU 104, DSP 106, connectivity block 110 (which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi- Fi connection, USB connection, Bluetooth connection, etc.), and the multimedia processor 112 (eg, it can detect and recognize gestures). In one implementation, the NPU is implemented in CPU 102 , DSP 106 and/or GPU 104 . The SOC 100 may also include a sensor processor 114 , an image signal processor (ISP) 116 and/or a navigation module 120 (which may include a global positioning system).

SOC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into the CPU 102 may include instructions for searching a stored multiplication result in a look-up table (LUT) corresponding to the multiplication product of the input value and the filter weight. code. The instructions loaded into CPU 102 may also include code for disabling the multiplier during the multiplication operation of the multiplication product when a multiplication product lookup table hit is detected. Additionally, the instructions loaded into the CPU 102 may include code for storing the calculated multiplicative product of the input value and the filter weight when a missing lookup table of the multiplicative product is detected.

SOC 100 and/or components thereof may be configured to perform video compression and/or decompression (also referred to as video encoding and/or decoding, which is referred to as collectively referred to as video decoding). Aspects of this subject matter can improve the efficiency of on-device video compression and/or decompression by using deep learning architectures to perform video compression and/or decompression. For example, a device using the video decoding techniques described can compress video more efficiently using machine learning-based techniques, the compressed video can be sent to another device, and the other device can use the machine-based Learned techniques to more efficiently decompress compressed video.

As previously mentioned, a neural network is an example of a machine learning system and can include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes of the input layer, processing is performed by hidden nodes of one or more hidden layers, and output is generated via output nodes of the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network can include a feature map or activation map that can include artificial neurons (or nodes). Feature maps may include filters, kernels, etc. The nodes may include one or more weights to indicate the importance of the nodes of one or more of the layers. In some cases, deep learning networks can have a series of multiple hidden layers, where early layers are used to determine simple and low-level features of the input, and later layers build up a hierarchy of more complex and abstract features.

Deep learning architectures can learn hierarchies of features. For example, if provided with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, the first layer can learn to recognize spectral power in specific frequencies if provided with auditory data. A second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual material or combinations of sounds for auditory material. For example, higher layers can learn to represent complex shapes in visual material or words in auditory material. Higher layers can learn to recognize common visual objects or spoken phrases.

Deep learning architectures may perform especially well when applied to problems with a natural hierarchy. For example, classifying motor vehicles could benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher layers to recognize cars, trucks, and planes.

Neural networks can be designed with a variety of connectivity patterns. In a feed-forward network, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As previously mentioned, hierarchical representations can be built in successive layers of a feed-forward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be sent to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one chunk of input data that is delivered sequentially to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be elucidative when identifying high-level concepts can aid in identifying specific low-level features of the input.

The connections between the layers of the neural network can be fully connected or locally connected. FIG. 2A shows an example of a fully connected neural network 202 . In a fully connected neural network 202, a neuron in the first layer can send its output to every neuron in the second layer, so that every neuron in the second layer will receive every neuron from the first layer. One is the input of the neuron. FIG. 2B shows an example of a locally connected neural network 204 . In a locally connected neural network 204, neurons in a first layer can be connected to a limited number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured such that each neuron in a layer will have the same or similar connection pattern, but with possibly different values (e.g., 210, 212 , 214 and 216) connection strength. Linear patterns of local connections can result in spatially distinct receptive fields in higher layers, since higher layer neurons in a given region can receive images tuned to a restricted fraction of the network's total input via training. attribute input.

An example of a locally connected neural network is a convolutional neural network. FIG. 2C shows an example of a convolutional neural network 206 . Convolutional neural network 206 may be configured such that connection strengths associated with inputs to each neuron of the second layer are shared (eg, 208 ). Convolutional neural networks can be well suited for problems where the spatial location of the input is meaningful. According to various aspects of the subject matter, convolutional neural network 206 may be used to perform one or more aspects of video compression and/or decompression.

One type of convolutional neural network is a deep convolutional network (DCN). FIG. 2D shows a detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capture device 230 such as an onboard camera. The DCN 200 of the present example may be trained to recognize traffic signs and the digits provided on the traffic signs. Of course, the DCN 200 can be trained for other tasks, such as recognizing lane markings or recognizing traffic lights.

The DCN 200 may be trained using supervised learning. During training, DCN 200 may be presented with imagery, such as imagery 226 of a speed limit sign, and then a forward pass may be computed to produce output 222 . DCN 200 may include a feature extraction part and a classification part. Upon receiving image 226 , convolution layer 232 may apply a convolution kernel (not shown) to image 226 to generate first set 218 of feature maps. As an example, the convolution kernel used for the convolution layer 232 may be a 5x5 kernel that produces a 28x28 feature map. In this example, because four different feature maps are produced in the first set of feature maps 218 , four different convolution kernels are applied to image 226 at convolution layer 232 . A convolution kernel may also be called a filter or a convolution filter.

The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to produce the second set of feature maps 220 . The max pooling layer reduces the size of the first set 218 of feature maps. That is, the size of the second set 220 of feature maps (such as 14x14) is smaller than the size of the first set of feature maps 218 (such as 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

In the example of FIG. 2D , the second set of feature maps 220 is convolved to produce a first feature vector 224 . In addition, the first eigenvector 224 is further convolved to generate the second eigenvector 228 . Each feature of the second feature vector 228 may include a number corresponding to a possible feature of the image 226 such as "sign", "60", and "100". A softmax function (not shown) can convert the digits in the second feature vector 228 into probabilities. As such, the output 222 of the DCN 200 is the probability that the image 226 includes one or more features.

In this example, the probabilities for "symbol" and "60" in output 222 are higher than other items in output 222 (such as "30", "40", "50", "70", "80", " 90” and “100”). Before training, the output 222 produced by the DCN 200 may be incorrect. Therefore, the error between the output 222 and the target output can be calculated. The target output is the ground truth of the image 226 (eg, "symbol" and "60"). Subsequently, the weights of the DCN 200 can be adjusted so that the output 222 of the DCN 200 more closely aligns with the target output.

To adjust the weights, the learning algorithm may compute a gradient vector for the weights. Gradients may indicate as to how much the error will increase or decrease if the weights are adjusted. At the top layer, the gradient may correspond directly to the value of the weights connecting the firing neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient can depend on the values of the weights and the calculated error gradients of the higher layers. The weights can then be adjusted to reduce the error. This way of adjusting weights can be called "backpropagation" because it involves a "backward pass" through the neural network.

In practice, the error gradient of the weights can be computed over a small number of instances such that the computed gradient is close to the true error gradient. This approximation method can be called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the overall system has stopped decreasing or until the error rate has reached a target level. After learning, the DCN may be presented with new images, and a forward pass through the network may produce an output 222 that may be considered the DCN's inference or prediction.

Deep Belief Networks (DBNs) are probabilistic models that include multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. A DBN can be obtained by stacking layers of a Restricted Boltzmann Machine (RBM). RBM is a type of artificial neural network that can learn a probability distribution over a set of inputs. RBMs are often used in unsupervised learning because they can learn a probability distribution without information about the class each input should be classified into. Using a mixed unsupervised and supervised paradigm, the bottom RBM of a DBN can be trained in an unsupervised manner and can act as a feature extractor, and the top RBM can be trained in a supervised manner (based on the input and target categories from the previous layer joint distribution) and can act as a classifier.

A deep convolutional network (DCN) is a network of convolutional networks configured with additional pooling and non-linear (eg, regularization) layers. DCN has achieved state-of-the-art performance on many tasks. A DCN can be trained using supervised learning (where both the input and output targets are known for many examples) and the weights of the network are modified by using gradient descent methods.

A DCN may be a feed-forward network. Furthermore, as before, connections from a neuron in the first layer of a DCN to a set of neurons in the next higher layer are shared across neurons in the first layer. DCN's feed-forward and shared connections can be used for fast processing. For example, the computational burden of a DCN can be much smaller than that of a similarly sized neural network that includes recurrent or feedback connections.

The processing of each layer of a convolutional network can be thought of as a spatially invariant template or base projection. If the input is first decomposed into channels (such as the red, green, and blue channels of a color image), a convolutional network trained on that input can be considered three-dimensional, with two spatial dimensions along the axis, and the third dimension captures color information. The output of the convolutional connection can be thought of as forming a feature map in subsequent layers, where each element of the feature map (e.g., 220) receives input from a series of neurons in the previous layer (e.g., feature map 218) and from multiple channels input for each of the channels. Values in the feature map can be further processed with non-linearities such as rectified max(0,x). Values from neighboring neurons can be further pooled (this corresponds to downsampling) and can provide additional local invariance and dimensionality reduction.

FIG. 3 is a block diagram illustrating an example of a deep convolutional network 350 . The deep convolutional network 350 may include multiple different types of layers based on connectivity and weight sharing. As shown in FIG. 3 , the deep convolution network 350 includes a convolution block 354A and a convolution block 354B. Each of the convolution block 354A, the convolution block 354B may be configured with a convolution layer (CONV) 356 , a normalization layer (LNorm) 358 and a max pooling layer (MAX POOL) 360 .

The convolutional layer 356 may include one or more convolutional filters that may be applied to the input data 352 to generate a feature map. Although only two convolution blocks 354A, 354B are shown, the present disclosure is not so limited, but rather, any number of convolution blocks (e.g., block 354A, block 354B) may be included in the depth convolution block according to design preference. Network 350. Normalization layer 358 may normalize the output of the convolution filter. For example, normalization layer 358 may provide whitening or lateral suppression. A max pooling layer 360 may provide downsampled aggregation in space for local invariance and dimensionality reduction.

For example, parallel filter banks of deep convolutional networks can be loaded on CPU 102 or GPU 104 of SOC 100 to achieve high performance and low power consumption. In an alternate embodiment, the parallel filter banks may be loaded on the DSP 106 or the ISP 116 of the SOC 100 . In addition, deep convolutional network 350 may access other processing blocks that may be present on SOC 100 , such as sensor processor 114 and navigation module 120 dedicated to sensors and navigation, respectively.

Deep convolutional network 350 may also include one or more fully connected layers, such as layer 362A (labeled "FC1") and layer 362B (labeled "FC2"). The deep convolutional network 350 may further include a logistic regression (LR) layer 364 . Between each layer 356 , 358 , 360 , 362A, 362B, 364 of the deep convolutional network 350 are weights (not shown) to be updated. The output of each of these layers (e.g., 356, 358, 360, 362A, 362B, 364) may serve as Subsequent layers of input to learn hierarchical feature representations from input data 352 (eg, image, audio, video, sensor data, and/or other input data) provided at the beginning of convolution block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352 . Classification score 366 may be a set of probabilities, where each probability is a probability of including a feature of the input material from the set of features.

As mentioned above, digital video data can include large amounts of data, which can place a significant burden on communication networks and equipment that process and store the video data. For example, recording uncompressed video content typically results in large file sizes that increase substantially as the resolution of the recorded video content increases. In one illustrative example, uncompressed The 16-bit-per-channel video may occupy 12.4 megabytes per frame or 297.6 megabytes per second. Uncompressed 16-bit-per-channel video recorded at 4K resolution at 24 frames per second may occupy 49.8 megabytes per frame or 1195.2 megabytes per second.

Network bandwidth is another constraint where large video files can become problematic. For example, video content is often delivered over wireless networks (e.g., via LTE, LTE-Advanced, New Radio (NR), WiFi ^™ , Bluetooth ^™ , or other wireless networks) and may constitute a significant portion of consumer Internet traffic. big part. Despite advances in the amount of bandwidth available in wireless networks, it may still be desirable to reduce the amount of bandwidth used to deliver video content in these networks.

Since uncompressed video content can result in large files, which can involve considerable memory for physical storage and considerable bandwidth for transmission, video decoding techniques can be used for compression and subsequent decompression such video content.

In order to reduce the size of video content—and thus reduce the amount of storage involved in storing video content and the amount of bandwidth involved in transmitting video content, it is possible to reduce the size of video content based on certain video coding standards (such as HEVC, AVC, MPEG, VVC etc.) to perform video decoding techniques. Video decoding usually uses predictive methods (such as inter-frame prediction, intra-frame prediction, etc.), which exploit the redundancy that exists in video images or sequences. A common goal of video decoding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing video quality degradation. As the demand for video services grows and new video services become available, coding techniques with better coding efficiency, performance, and rate control are required.

Generally speaking, the encoding device encodes the video data according to the video decoding standard to generate an encoded video bit stream. In some examples, an encoded video bitstream (or "video bitstream" or "bitstream") is a series of one or more encoded video sequences. An encoding device may generate an encoded representation of a picture via partitioning each picture into slices. A slice is independent of other slices such that information in that slice is encoded without relying on data from other slices within the same picture. A slice includes one or more slice segments, including independent slice segments and, if present, one or more dependent slice segments that depend on previous slice segments. In HEVC, a slice is then partitioned into coding tree blocks (CTBs) of luma samples and chroma samples. A CTB of luma samples and one or more CTBs of chroma samples, together with the syntax for the samples, are called a coding tree unit (CTU). A CTU may also be called a "treeblock" or a "largest coding unit" (LCU). CTU is the basic processing unit for HEVC encoding. A CTU can be split into multiple coding units (CUs) of different sizes. A CU contains an array of luma and chroma samples called a coding block (CB).

Luma and chroma CBs can be further split into prediction blocks (PBs). A PB is a block of samples of luma or chroma components that use the same motion parameters for inter-frame prediction or intra-block copy (IBC) prediction when available or enabled for use. A luma PB and one or more chroma PBs, along with associated syntax, form a prediction unit (PU). For inter-frame prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) is signaled in the bitstream for each PU and used for luma PB and one or more Inter-frame prediction of chroma PB. The exercise parameters can also be referred to as exercise information. A CB can also be partitioned into one or more transform blocks (TB). A TB represents a square block of samples of a color component to which a residual transform (eg, in some cases, the same two-dimensional transform) is applied to code the prediction residual signal. A transform unit (TU) represents a TB of luma and chroma samples and corresponding syntax elements. Transform coding is described in more detail below.

According to the HEVC standard, transformation can be performed using TUs. The size of a TU may be set based on the size of the PUs within a given CU. A TU can have the same size as a PU or be smaller than a PU. In some instances, a quadtree structure known as a residual quadtree (RQT) may be used to subdivide residual samples corresponding to a CU into smaller units. A leafline node of an RQT may correspond to a TU. The primitive delta values associated with a TU may be transformed to generate transform coefficients. The transform coefficients may then be quantized by an encoding device.

Once a picture of video data is partitioned into CUs, the encoding device uses prediction modes to predict each PU. The prediction units or prediction blocks are then subtracted from the original video data to obtain residuals (described below). For each CU, the prediction mode can be signaled within the bitstream using syntax data. The prediction mode may include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction exploits the correlation between spatially adjacent samples within a picture. For example, using intra prediction, each PU is predicted from neighboring image data in the same picture, using e.g. DC prediction to find the mean value for the PU, using planar prediction to fit the planar surface to the PU, Use directional predictions to extrapolate from neighboring data, or use any other suitable prediction type. Inter-frame prediction uses the temporal correlation between pictures to derive a motion-compensated prediction for a block of image samples. For example, using inter-frame prediction, each PU is predicted from image data in one or more reference pictures (before or after the current picture in output order) using motion compensated prediction. For example, a decision on whether to use inter-picture prediction or intra-picture prediction to decode a picture region can be made at the CU level.

After performing prediction using intra prediction and/or inter prediction, the encoding device may perform transform and quantization. For example, after prediction, the encoding device may calculate a residual value corresponding to the PU. Residual values may include primitive difference values between the current block (PU) being encoded and a prediction block used to predict the current block (eg, a predicted version of the current block). For example, after generating a prediction block (eg, performing inter prediction or intra prediction), the encoding apparatus may generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of primitive difference values that quantize the difference between the primitive values of the current block and the primitive values of the predicted block. In some examples, a residual block may be represented in a two-dimensional block format (eg, a two-dimensional matrix or array of primitive values). In such instances, a residual block is a two-dimensional representation of primitive values.

Any residual data that may remain after performing prediction is transformed using a block transform, which may be based on discrete cosine transform, discrete sine transform, integer transform, wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (eg, of size 32x32, 16x16, 8x8, 4x4, or other suitable size) may be applied to the residual data in each CU. In some embodiments, TUs may be used for transform and quantization procedures implemented by encoding devices. A given CU having one or more PUs may also include one or more TUs. As described in further detail below, residual values may be transformed into transform coefficients using a block transform, and then quantized and scanned using TUs to produce serialized transform coefficients for entropy coding.

The encoding device may perform quantization of transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient having an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

Once quantization is performed, the encoded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction mode, motion vector, block vector, etc.), segmentation information, and any other appropriate data (such as other syntax data ). The different elements of the encoded video bitstream can then be entropy encoded by the encoding device. In some examples, an encoding device may scan quantized transform coefficients using a predefined scan order to generate serialized vectors that may be entropy encoded. In some examples, encoding devices may perform self-adjusting scans. After scanning the quantized transform coefficients to form a vector (eg, a one-dimensional vector), the encoding device may entropy encode the vector. For example, the encoding device may use context self-adjusting variable length coding, context self-adjusting binary arithmetic coding, syntax-based context self-adjusting binary arithmetic coding, probability interval partitioning entropy coding, or another suitable entropy coding technique .

The encoding device may store the encoded video bitstream and/or may send the encoded video bitstream data over a communication link to a receiving device (which may include a decoding device). The decoding device may decode the encoded video bitstream data via entropy decoding (eg, using an entropy decoder) and extracting elements of one or more encoded video sequences constituting the encoded video data. The decoding device may then rescale and inverse transform the encoded video bitstream data. The residual data is then passed to the prediction stage of the decoding device. The decoding device then predicts the block (eg, PU) using intra prediction, inter prediction, IBC, and/or other types of prediction. In some instances, the predictions are summed with the output of the inverse transformation (residual data). The decoding device may output the decoded video to a video destination device, which may include a display or other output device for displaying the decoded video data to a consumer of the content.

Video coding systems and techniques defined by various video coding standards (e.g., the above-mentioned HEVC video coding technique) may be able to preserve most of the information in the original video content, and are a priori defined based on signal processing and information theory concepts of. However, in some cases, machine learning (ML) based image and/or video systems can provide superior performance over non-ML based image and video decoding systems such as end-to-end neural network based image and video decoding ( E2E-NNVC) system) advantages. As mentioned earlier, many E2E-NNVC systems are designed as an autoencoder subnetwork (encoder subnetwork) and a second subnetwork responsible for learning a probability model on quantized latent variables for entropy decoding combination. This architecture can be viewed as a combination of a transform plus quantization module (encoder subnetwork) and an entropy modeling subnetwork module.

FIG. 4 illustrates a system 400 that includes a device 402 configured to perform video encoding and decoding using an E2E-NNVC system. Device 402 is coupled to camera 407 and storage medium 414 (eg, a data storage device). In some implementations, the camera 407 is configured to provide image data 408 (eg, a video data stream) to the processor 404 for encoding by the E2E-NNVC system. In some implementations, device 402 can be coupled to and/or can include multiple cameras (eg, a two-camera system, three cameras, or other number of cameras). In some cases, device 402 may be coupled to a microphone and/or other input device (e.g., a keyboard, mouse, touch input device such as a touch screen and/or trackpad, and/or other input device) . In some examples, camera 407 , storage medium 414 , microphone, and/or other input devices may be part of device 402 .

The device 402 is also coupled to a second device 490 via a transmission medium 418 (eg, one or more wireless networks, one or more wired networks, or a combination thereof). For example, transmission medium 418 may include a channel provided by a wireless network, a wired network, or a combination of wired and wireless networks. Transmission medium 418 may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. Transmission media 418 may include routers, switches, base stations, or any other device that may be used to facilitate communication from a source device to a receiving device. A wireless network may include any wireless interface or combination of wireless interfaces, and may include any suitable wireless network (e.g., the Internet or other wide area network, packet-based networking, WiFi ^™ , radio frequency (RF), UWB, WiFi Direct Connect, Cellular, Long Term Evolution (LTE), WiMax ^TM , etc.). A wired network may include any wired interface (eg, fiber optics, Ethernet, Ethernet over power line, Ethernet over coaxial cable, Digital Signal Line (DSL), etc.). Wired and/or wireless networks can be implemented using various devices, such as base stations, routers, access points, bridges, gateways, switches, and so on. The encoded video bit stream data can be modulated according to a communication standard such as a wireless communication protocol and sent to a receiving device.

Device 402 includes one or more processors (referred to herein as "processors") 404 coupled to memory 406, a first interface ("I/F 1") 412, and a second interface ("I/F 2") ”)416. Processor 404 is configured to receive image data 408 from camera 407 , from memory 406 and/or from storage medium 414 . Processor 404 is coupled to storage medium 414 via a first interface 412 (e.g., via a memory bus), and via a second interface 416 (e.g., network peripherals, wireless transceivers and antennas, one or more other network peripheral devices, or a combination thereof) is coupled to transmission medium 418 .

Processor 404 includes E2E-NNVC system 410 . The E2E-NNVC system 410 includes an encoder portion 462 and a decoder portion 466 . In some implementations, the E2E-NNVC system 410 may include one or more autoencoders. Encoder portion 462 is configured to receive input data 470 and process input data 470 to generate output data 474 based at least in part on input data 470 .

In some implementations, encoder portion 462 of E2E-NNVC system 410 is configured to perform anamorphic compression on input material 470 to generate output material 474 such that output material 474 has fewer bits than input material 470 . The encoder portion 462 can be trained to compress input data 470 (eg, images or video frames) without using motion compensation based on any previous representation (eg, one or more previously reconstructed frames). For example, encoder portion 462 may only use video data from a video frame to compress the video frame without using any data from previously reconstructed frames. The video frames processed by the encoder portion 462 may be referred to herein as intra-frame predicted frames (I frames). In some examples, the I-frames may be generated using conventional video coding techniques (eg, according to HEVC, VVC, MPEG-4, or other video coding standards). In such examples, the processor 404 may include or be coupled to a video coding device (e.g., an encoding device) configured to perform block-based intra prediction, such as the intra prediction described above with respect to the HEVC standard. . In such instances, processor 404 may not include E2E-NNVC system 410 .

In some implementations, the encoder portion 462 of the E2E-NNVC system 410 can be trained to compress the input data 470 (e.g., video frame). For example, encoder portion 462 may use video data from a video frame and use data from a previously reconstructed frame to compress a video frame. The video frames processed by the encoder portion 462 may be referred to herein as intra-frame predicted frames (P frames). Motion compensation can be used to determine the data of the current frame by describing how the picture elements from the previously reconstructed frame moved to the new position in the current frame and the residual information.

As depicted, the encoder portion 462 of the E2E-NNVC system 410 includes a neural network 463 and a quantizer 464 . Neural network 463 may include one or more convolutional neural networks (CNNs), one or more fully connected neural networks, one or more gated recurrent units (GRUs), one or more long Short-term memory (LSTM) networks, one or more convolutional RNNs, one or more convolutional GRUs, one or more convolutional LSTMs, one or more GANs, any combination thereof, and/or other types of neural network architectures. Intermediate data 472 is input to quantizer 464 . Examples of components that may be included in the encoder portion 462 are illustrated in FIGS. 6A-6E .

Quantizer 464 is configured to perform quantization and, in some cases, entropy coding of intermediate material 472 to produce output material 474 . Output data 474 may include quantized (and, in some cases, entropy encoded) data. The quantization operation performed by quantizer 464 may result in the generation of quantized codes from intermediate data 472 (or data representing quantized codes generated by E2E-NNVC system 410 ). Quantization codes (or data representing quantization codes) may also be called latent codes or latent variables (denoted as z ). The entropy model applied to the latent variables may be referred to herein as a "prior". In some examples, quantization and entropy coding operations may be performed using existing quantization and/or entropy coding operations performed when encoding and/or decoding video data according to existing video coding standards. In some examples, quantization and/or entropy coding operations may be performed by the E2E-NNVC system 410 . In one illustrative example, E2E-NNVC system 410 may be trained using supervised training, where during training residual data is used as input and quantization and entropy codes are used as known outputs (labels).

Decoder portion 466 of E2E-NNVC system 410 is configured to receive output data 474 (eg, directly from quantizer 464 and/or from storage medium 414 ). Decoder portion 466 may process output data 474 to generate representation 476 of input data 470 based at least in part on output data 474 . In some examples, decoder portion 466 of E2E-NNVC system 410 includes neural network 468, which may include one or more CNNs, one or more fully connected neural networks, one or more GRUs, one or more Long short-term memory (LSTM) network, one or more convolutional RNNs, one or more convolutional GRUs, one or more convolutional LSTMs, one or more GANs, any combination thereof, and/or other types of neural network architectures . Examples of components that may be included in decoder portion 466 are shown in FIGS. 6A-6E .

Processor 404 is configured to send output data 474 to at least one of transmission medium 418 or storage medium 414 . For example, output data 474 may be stored at storage medium 414 for later retrieval and decoding (or decompression) by decoder portion 466 to produce representation 476 of input data 470 as reconstructed data. The reconstructed data may be used for various purposes, such as for replaying video data that has been encoded/compressed to generate output data 474 . In some implementations, output material 474 can be decoded at another decoder device matched to decoder portion 466 (e.g., in device 402, in second device 490, or in another device) to produce a pair of A representation 476 of data 470 is input as reconstructed data. For example, second device 490 may include a matching (or substantially matching) decoder to decoder portion 466 and may transmit output material 474 to second device 490 via transmission medium 418 . The second device 490 may process the output data 474 to produce a representation 476 of the input data 470 as reconstructed data.

Components of system 400 may include and/or may be implemented using electrical circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units ( GPU), digital signal processor (DSP), central processing unit (CPU), and/or other appropriate electronic circuits), and/or may include computer software, firmware, or any combination thereof, and/or use them to implement , to perform the various operations described in this article.

Although system 400 is shown as including certain components, one of ordinary skill in the art to which this invention pertains will appreciate that system 400 may include more or fewer components than those shown in FIG. 4 . For example, system 400 may also include input devices and output devices (not shown), or may be part of a computing device that includes input devices and output devices. In some implementations, system 400 may also include, or may be part of, a computing device that includes: one or more memory devices (e.g., one or more random access memory (RAM) components , read-only memory (ROM) components, cache memory components, buffer components, database components, and/or other memory devices), communicate with and/or be electrically connected to one or more memory devices one or more processing devices (e.g., one or more CPUs, GPUs, and/or other processing devices), one or more wireless interfaces for performing wireless communications (e.g., including a or multiple transceivers and baseband processors), one or more wired interfaces for performing communications over one or more hardwired connections (e.g., serial interfaces such as Universal Serial Bus (USB) inputs, lighting connector, and/or other wired interface), and/or other components not shown in FIG. 4 .

In some implementations, system 400 can be implemented locally by and/or included in a computing device. For example, computing devices may include mobile devices, personal computers, tablet computers, virtual reality (VR) devices (e.g., head-mounted displays (HMDs) or other VR devices), augmented reality (AR) devices (e.g., HMDs, AR glasses or other AR devices), wearable devices, servers (e.g., in a software-as-a-service (SaaS) system or other server-based systems), televisions, and/or any other computing device.

In one example, the E2E-NNVC system 410 may be incorporated into a portable electronic device comprising: a memory 406 coupled to the processor 404 and configured to store instructions executable by the processor 404; and a wireless transceiver coupled to the antenna and processor 404 and operable to transmit output data 474 to a remote device.

E2E-NNVC systems are generally designed to handle RGB input. Examples of image and video decoding schemes for RGB input are described in: J.Balle, D.Minnen, S.Singh, S.J.Hwang, N.Johnston, "Variational image compression with a scale hyperprior", ICLR, 2018 (referred to as "J.Balle paper"), and D.Minnen, J.Balle, G.Toderici, "Joint Autoregressive and Hierarchical Priors for Learned Image Compression", CVPR 2018 (referred to as "D.Minnen paper" ), which is hereby incorporated by reference in its entirety for all purposes.

Fig. 5 is a schematic diagram showing an example of the E2E-NNVC system described in the J. Balle paper. The g _a and g _s subnetworks in the E2E-NNVC system of FIG. 5 correspond to the encoder subnetwork (eg, encoder part 462 ) and decoder subnetwork (eg, decoder part 466 ), respectively. The g _a and g _s subnetworks of Figure 5 are designed for a three-channel RGB input, where all three R, G, and B input channels go through the same neural network layers (convolutional layers and generalized division normalization (GDN) layer) and are handled by it. Neural network layers may include convolutional layers that perform convolutional operations and GDN and/or inverse GDN (IGDN) nonlinear layers that implement local division regularization. Local division regularization is a type of transformation that has been shown to be particularly suitable for density modeling and compression of imagery. E2E-NNVC systems (such as those shown in Figure 5) target input channels with similar statistical properties, such as RGB material (where the statistical properties of different R, G, and B channels are similar).

Although E2E-NNVC systems are typically designed to handle RGB input, most image and video coding systems use a YUV input format (eg, in many cases, YUV420 input format). The chroma (U and V) channels of data in YUV format may be subsampled relative to the luma (Y) channel. Subsampling has minimal impact on visual quality (eg, no noticeable or noticeable impact on visual quality). Subsampling formats include YUV420 format, YUV422 format, and/or other YUV formats. In YUV format, the correlation between channels is reduced, which may be different from other color formats (for example, RGB format). Also, the statistics for the luma (Y) and chroma (U and V) channels are different. For example, the U and V channels have less variance than the luma channel, while in eg RGB formats the statistical properties of the different R, G and B channels are more similar. Video transcoder - a decoder (or transcoder) is designed according to the input characteristics of the data (for example, a transcoder can encode and/or decode data according to the input format of the data). For example, if the chroma channel of a frame is subsampled (e.g., the resolution of the chroma channel is half that of the luma channel), then when the transcoder predicts the blocks of the frame for motion compensation, the width of the luma block and height will both be twice the size of the chroma block. In another example, a transcoder may decide how many primitives to encode or decode for chroma and luma, and so on.

If RGB input data (which, as mentioned, most E2E-NNVC systems are designed to process) is replaced by YUV 4:4:4 input data (where all channels have the same size), then due to the luma (Y) and chrominance (U And V) channel different statistical properties, E2E-NNVC system performance reduction in processing input data. As before, the chroma (U and V) channels are subsampled in some YUV formats, such as in the case of YUV420. For example, for content with a YUV 4:2:0 format, the U and V channels are half the resolution of the Y channel (due to half the width and height, the U and V channels are one quarter the size of the Y channel) . Such subsampling may result in incompatibility of the input data with that of the E2E-NNVC system. The input data is information that the E2E-NNVC system attempts to encode and/or decode (eg, a YUV frame comprising three channels including luma (Y) and chrominance (U and V) channels). Many neural network-based systems assume that all channels of input data are of the same size, and thus feed all input channels to the same network. In such cases, the outputs of some operations can be added (for example, using matrix addition), in which case the channels must be of the same size.

In some instances, to address such issues, the Y channel may be subsampled into four half-resolution Y channels. Four half-resolution Y channels can be combined with two chroma channels to form six input channels. Six input channels can be input or fed to the E2E-NNVC system designed for RGB input. Such methods can resolve issues regarding resolution differences between luma (Y) and chroma (U and V) channels. However, inherent differences between luma (Y) and chroma (U and V) channels still exist, resulting in poor coding (eg, encoding and/or decoding) performance.

As previously described, described herein are systems and techniques for performing image and/or video coding using one or more ML-based systems. The systems and techniques described herein provide front-end architectures (e.g., new subnetworks, such as those based on End-to-End Neural Networks for Image and Video Coding (E2E-NNVC) systems). In some instances, the front-end architecture is configured to accommodate YUV 4:2:0 input format in E2E-NNVC designed for RGB input format. As previously mentioned, the front-end architecture is applicable to many E2E-NNVC architectures (including, for example, the architectures described in the J.Balle paper and the D.Minnen paper). The systems and techniques described herein take into account the different characteristics of the luma (Y) and chroma (U and V) channels and the resolution differences of the luma (Y) and chroma (U and V) channels. The E2E-NNVC system can encode and/or decode individual frames (or images) and/or video data including multiple frames.

In some examples, the systems and techniques described herein can initially input or feed the Y and UV channels into two separate layers. Subsequently, the E2E-NNVC system can pair Y and UV channels after a certain number of layers (e.g., after the first pair of convoluted and nonlinear layers or other layers, as shown in Figures 6A-6E described below). Associated data are combined. Since the U and V chroma components are subsampled relative to the luma (Y) channel, the subsampling in the first convoluted layer can be skipped, and a certain size (e.g., has a value of (N/2+1)x( A convoluted (eg, CNN) core of size N/2+1) can be used to subsample the input chroma (U and V) channels. Then a CNN core with a different size (eg, NxN CNN cores) can be used for the luma (Y) channel than the cores used for the chroma (U and V) channels. The two branches of the front-end architecture (carrying luma and chroma channel or component information, respectively) can be combined using a convolutional layer (eg, a 1x1 convolutional layer) that combines values across channels. The use of 1x1 convolutional layers can provide various benefits as described herein, including improved decoding efficiency.

6A-6F illustrate an example of a front-end architecture of a neural network system. In some examples, the front-end architecture of FIGS. 6A-6F may be part of an E2E-NNVC system designed to process (encode and/or decode) data in YUV 4:2:0 format. For example, the front-end architecture may be configured to process input data in YUV 4:2:0 format. The front-end architectures of Figures 6A, 6C, 6D and 6E apply two different non-linear operators after the 1x1 convoluted layer. For example, the generalized divisor normalization (GDN) operator is used in the architecture of FIG. 6A, while the parameter rectified linear unit (PReLU) nonlinear operator is applied in the architecture of FIGS. 6C-6E. In some examples, similar neural network architectures as shown in FIG. 6A and FIGS. etc.) and/or content with other input formats to encode and/or decode.

For example, FIG. 6A is a schematic diagram illustrating an example of a front-end neural network system or architecture that can be configured to work directly with 4:2:0 input (Y, U, and V) data. As shown in Figure 6A, at the encoder sub-network of the neural network system, a 1x1 convolutional layer 606 is used to split the luma and chroma channels (luma (Y) channel 602 and chroma (U and V) channels 604) Combinations are made and then GDN nonlinear operators 608 are applied. Similar operations are performed on the decoder subnetwork of the neural network system, but in reverse order. For example, as shown in Figure 6A, applying an inverse GDN (IGDN) operator 609, a 1x1 convoluted layer 613 is used to separate the Y and U, V channels, and the corresponding IGDN 615, IGDN 616 and convoluted layers 617, 618 are used to separate Handle separate Y channels and U, V channels.

For example, the first two neural network layers in the encoder sub-network of the neural network system of FIG. 6A include a first convolutional layer 611 (denoted as Ncov|3x3)|↓1), a second convolutional layer 610 (denoted as Nconv | 5x5|↓2), the first GDN layer 614 and the second GDN layer 612 . The last two neural network layers in the decoder sub-network of the front-end neural network architecture of FIG. 6A include a first inverse GDN (IGDN) layer 616, a second inverse GDN (IGDN) 615, an A first convoluted layer 618 (denoted as 2conv|3x3)|↑1) of reconstructed chroma (U and V) components, and a second convolutional layer used to generate the reconstructed luma (Y) component of the frame 617 (expressed as 1conv|5x5)|↑2). The "Nconv" notation refers to the number of output channels (N) (corresponding to the number of output features) of a given convolutional layer (where the N value defines the number of output channels). The 3x3 and 5x5 notation indicates the size of the corresponding convolution core (eg, 3x3 core and 5x5 core). The "↓1" and "↓2" notations refer to span values, where ↓1 refers to a span of 1 (for downsampling, as indicated by "↓"), and ↓2 refers to a span of 1 pitch (for downsampling). The "↑1" and "↑2" notations refer to stride values, where ↑1 refers to a stride of 1 (for upsampling, as indicated by "↑"), and ↑2 refers to a stride of 1 pitch (for upsampling).

For example, the convolutional layer 610 downsamples the input luma channel 602 by a factor of 4 by applying a 5x5 convolutional filter with a stride value of 2 in the horizontal and vertical dimensions. The resulting output of the convolutional layer 610 is an array of N eigenvalues (corresponding to N channels). The convolutional layer 611 processes the input chroma (U and V) channels 604 by applying a 3x3 convolutional filter with a stride value of 1 in the horizontal and vertical dimensions. The resulting output of the convolution layer 611 is an array of N eigenvalues (corresponding to N channels). The eigenvalue array output by the convolution layer 610 has the same size as the eigenvalue array output by the convolution layer 611 . Subsequently, the GDN layer 612 may process the feature values output by the convolution layer 610 , and the GDN layer 614 may process the feature values output by the convolution layer 611 .

The 1x1 convolution layer 606 may then process the feature values output by the GDN layer 612 , GDN layer 614 . The 1x1 convoluted layer 606 may produce a linear combination of features associated with the luma channel 602 and the chroma channel 604 . The linear combination operation operates as a per-value cross-channel mix of Y and UV components, resulting in cross-component (eg, cross-lumina and chroma) predictions that enhance coding performance. Each 1x1 convolution filter of 1x1 convolution layer 606 may include a respective scaling factor applied to a corresponding Nth channel of luma channel 602 and a corresponding Nth channel of chroma channel 604 .

FIG. 6B is a schematic diagram illustrating an example operation of the 1×1 convoluted layer 638 . As mentioned above, N represents the number of output channels. As shown in FIG. 6B , 2N channels are provided as input to a 1x1 convoluted layer 638 , including an N-channel chrominance (combined U and V) output 632 and an N-channel luminance (Y) output 634 . In the example of FIG. 6B , the value of N is equal to 2, which indicates two channels with an N-channel chroma output 632 value and two channels with an N-channel luma output 634 value. Referring to FIG. 6A , an N-channel chroma output 632 may be an output from the GDN layer 614 , and an N-channel luma output 634 may be an output from the GDN layer 612 . However, in other examples, N-channel chrominance output 632 and N-channel luma output 634 may be obtained from other nonlinear layers (e.g., from pReLU layer 652 and pReLU layer 654, respectively, of FIG. 6D , from pReLU layer 662 and pReLU layer, respectively, of FIG. pReLU layer 664 ) or directly from the convolutional layer (eg, from the convolutional layer 670 and the convolutional layer 671 of FIG. 6F , respectively).

The 1x1 convolutional layer 638 processes 2N channels and performs a linear combination of features of the 2N channels, and then outputs a set of N-channel features or coefficients. The 1x1 convolution layer 638 includes two 1x1 convolution filters (based on N=2). A first 1x1 convolution filter is shown with a value of s ₁ and a second 1x1 convolution filter is shown with a value of s ₂ . The s ₁ value represents the first scaling factor, and the s ₂ value represents the second scaling factor. In one illustrative example, the s ₁ value is equal to 3, and the s ₂ value is equal to 4. Each of the 1x1 convolution filters of the 1x1 convolution layer 638 has a stride value of 1, which indicates a scaling factor s ₁ and a scaling factor s ₂ will be applied to each of the UV output 632 and Y output 634 values.

For example, the scaling factor s ₁ of the first 1×1 convolution filter is applied to each value of the first channel ( C1 ) of the UV output 632 and each value of the first channel ( C1 ) of the Y output 634 . Once each value of the first channel (C1) of the UV output 632 and each value of the first channel (C1) of the Y output 634 is scaled by the scaling factor s ₁ of the first 1x1 convolution filter, the scaled The values of the first channel (C1) are combined into an output value of 639. The scaling factor s ₂ of the second 1×1 convolution filter is applied to each value of the second channel ( C2 ) of the UV output 632 and each value of the second channel ( C2 ) of the Y output 634 . After scaling each value of the second channel (C2) of the UV output 632 and each value of the second channel (C2) of the Y output 634 via the scaling factor s2 of the _second 1x1 convolution filter, the scaled The values of the second channel (C2) are combined into an output value of 639. Therefore, the four Y channels and UV channels (two Y channels and two combined UV channels) are mixed or combined into two output channels C1 and C2.

Returning to Figure 6A, the output of the 1x1 convolutional layer 606 is processed by an additional GDN layer and an additional convolutional layer of the encoder sub-network. The quantization engine 620 may perform quantization on the features output by the final neural network layer 619 of the encoder sub-network to produce a quantized output. The entropy encoding engine 621 may entropy encode the quantized output from the quantization engine 620 to generate a bitstream. As shown in FIG. 6A , the entropy coding engine 621 can perform entropy coding using the prior generated by the super prior network. A neural network system may output a bitstream for storage, for transmission to another device, server device or system, and/or otherwise output a bitstream.

The decoder subnetwork of the NNS or the decoder subnetwork of another NNS (another device) can decode the bitstream. For example, as shown in FIG. 6A , the entropy decoding engine 622 of the decoder sub-network can perform entropy decoding on the bitstream and output the entropy-decoded data to the dequantization engine 623 . The entropy decoding engine 622 may perform entropy decoding using the prior generated by the super prior network, as shown in FIG. 6A . The dequantization engine 623 can dequantize the material. The dequantized data can be processed by multiple convolutional layers and multiple inverse GDNs (IGDNs) in the decoder subnetwork.

After being processed by the IGDN layer 609, the 1x1 convolutional layer 613 can process the data. The 1x1 convolutional layer 613 may include 2N convolutional filters, which may divide the data into Y channel features and combined UV channel features. For example, each of the N channels output by the IGDN layer 609 may be processed using 2N 1x1 convolutions of the 1x1 convolution layer 613 (resulting in scaling). For each scaling factor _ni corresponding to an output channel applied to N input channels (2N output channels in total), the decoder subnetwork can perform summation over the N input channels, resulting in 2N outputs. In one illustrative example, for a scaling factor n ₁ , the decoder subnetwork may apply the scaling factor n ₁ to N input channels and may sum the results, which results in one output channel. The decoder sub-network may perform this operation for 2N different scaling factors (eg scaling factor n ₁ , scaling factor n ₂ to scaling factor n _2N ).

The Y channel features output by the 1x1 convoluted layer 613 can be processed by the IGDN 615 . The combined UV channel features output by the 1x1 convoluted layer 613 can be processed by the IGDN 616 . Convolutional layer 617 may process Y channel features and output a reconstructed Y channel per pixel or sample of the reconstructed frame (eg, luma samples or pixels), shown as reconstructed Y component 624 . The convoluted layer 618 may process combined UV channel features and may output a reconstructed U channel per primitive or sample of the reconstructed frame (e.g., chroma blue samples or primitives) and the reconstructed signal A frame's reconstructed V channel per primitive or sample (eg, chroma red samples or primitives), shown as reconstructed U and V components 625 .

6C is a schematic diagram illustrating another example of a front-end neural network system or architecture that can be configured to operate directly with 4:2:0 input (Y, U, and V) input data. As shown in Figure 6C, at the encoder subnetwork of the neural network system, a 1x1 convolutional layer 648 is used to combine the branch luma and chrominance channels (luminance channel 642 and chrominance channel 644) (similar to 6A ), and then apply the pReLU non-linear operator 649 . In other examples, non-linear operators other than pReLU non-linear operators may be applied. Similar operations are performed by the decoder subnetwork of the neural network system of Figure 6C (similar to that described above with respect to Figure 6A), but in reverse order (e.g., applying pReLU operands, using 1x1 convolutional layers to separate Y and U , V channel, and the separate Y and U, V channels are processed using the corresponding inverse IGDN and convoluted layers).

Compared with the E2E-NNVC system (neural network based transcoder) described in Fig. 5, via the first two network layers in g _a (encoder side) and corresponding g _s (decoder side) The input processing of the front-end architecture of FIGS. 6A and 6C is modified by separately processing the Y channel and the UV channel. The first convolutional layer (eg, convolutional layer 610 in FIG. 6A and convolutional layer 646 in FIG. 6C ) for processing the Y component (denoted as Ncov|5x5|↓2) can be compared with the first convolutional layer 510 in FIG. same or similar. Similarly, the second convoluted layer (denoted as 1conv|5x5|↑2) of the decoder sub-networks of Figures 6A and 6C to produce the reconstructed luminance (Y) component can be compared with that of the system of Figure 5 The last convolutional layer of the decoder subnetwork _gs is the same or similar. Unlike the system of Figure 5, the U and V chrominance channels are provided by the architectures of Figures 6A and 6C using separate convolutional layers (e.g., separate CNNs such as convolutional layer 611 in Figure 6A or convolutional layer 647 in Figure 6C) (Denoted as Ncov | 3x3 | ↓ 1), where the size of the core is half of the Y core of the Ncov | 5x5 | ↓ 2 convoluted layer 610 in FIG. 6A or the Ncov | 5x5 | (and no downsampling, corresponding to a stride equal to 1), followed by specific GDN layers (one GDN layer for luma Y, and one GDN layer for chroma U and V).

After the convoluted layers (the first pair of CNN Nconv|5x5|↓2 and Nconv|3x3|↓1 layers) and GDN layers in Figures 6A and 6C, the luma (Y) channel and chrominance (U and V) channels (e.g. , transformed or filtered versions of the input channels) representations or features are of the same size and are then combined using the 1x1 convolutional layer 606 of FIG. 6A or the 1x1 convolutional layer 648 of FIG. 6C. For example, in the YUV 4:2:0 format, the luma (Y) channel is twice as large in each dimension as the chroma (U and V) channels. When subsampling the chroma (U and V) channels via 2 pairs, the output produced based on processing these channels becomes the same dimensionality as the conv2d output of the luma channel (since the luma channel is not subsampled). The separate normalization of the channels accounts for the difference in variance of the luma and chroma channels. As before, non-linear operators can then be applied (eg, using GDN 608 or pReLU 649 ) before using the additional three convolutional layers until the quantization step is reached.

In the decoder subnetwork of the architecture in Figure 6A and Figure 6C, separate IGDN and Convolutional layers are used to separately generate the reconstructed luma (Y) component and the reconstructed chrominance (U and V) components . For example, the kernel size of the convoluted layer 618 (2conv|3x3|↑1 layer of the decoder subnet) used to generate the reconstructed chrominance (U and V) components 625 of FIG. Half the kernel size (and no upsampling, corresponding to a stride equal to 1) used in the convoluted layer 617 of the reconstructed luma (Y) component 624 (1conv|5x5|↑2 layer of the decoder subnetwork).

6D is a schematic diagram illustrating another example of a front-end neural network architecture that can be configured to operate directly with 4:2:0 input (Y, U, and V) input data. As shown in Figure 6D, at the encoder side, a 1x1 convoluted layer is used to combine the branched luma and chroma channels, and then the pReLU non-linear operator is applied. Compared to the architecture shown in Figure 6A and Figure 6C, the GDN layers in the luma and chrominance branches are replaced with PReLU operands.

6E is a schematic diagram illustrating another example of a front-end neural network architecture that can be configured to operate directly with 4:2:0 input (Y, U, and V) input data. As shown in Figure 6E, on the encoder side, the branch luma and chroma channels are combined using a 1x1 convoluted layer, and the pReLU non-linear operator is then applied. Compared with the architectures shown in FIGS. 6A , 6C and 6D, all GDN layers of the architecture in FIG. 6E are replaced with PReLU operands.

6F is a schematic diagram illustrating another example of a front-end neural network architecture that can be configured to operate directly with 4:2:0 input (Y, U, and V) input data. As shown in Figure 6F, on the encoder side, a 1x1 convoluted layer is used to combine the branch luma and chroma channels. Compared to the architecture shown in FIGS. 6A-6E , all GDN layers are completely removed, and no non-linear activation operation is used between convolutional layers.

The neural network architecture designs shown in FIGS. 6C-6F can be used to reduce GDN layers (e.g., as shown in the architecture of FIG. 6C ) or completely remove GDN layers (e.g., as shown in the architectures of FIGS. 6E and 6F ). ).

In some examples, the systems and techniques described herein can be used for other encoder-decoder subnetworks that use a combination of convolution (eg, CNN) and normalization stages at the input of a neural network based coding system.

7 is a flowchart illustrating an example of a process 700 for processing video using one or more machine learning techniques described herein. At block 702, the process 700 includes generating, by a first convolutional layer of an encoder sub-network of the neural network system, an output value associated with a luma channel of a frame.

At block 704, the process 700 includes generating, by a second convolutional layer of the encoder subnetwork, an output value associated with at least one chrominance channel of the frame. At block 706, process 700 includes generating, by the third convolutional layer, a combined representation of the frame based on output values associated with a luma channel of the frame and output values associated with at least one chrominance channel of the frame. In some cases, the third convolutional layer includes a 1x1 convolutional layer (eg, the 1x1 convolutional layer of the encoder subnetwork of FIGS. 6A-6F ) that includes one or more 1x1 convolutional filters. At block 708, the process 700 includes generating encoded video data based on the combined representation of the frames.

In some examples, process 700 includes processing an output value associated with a luma channel of a frame using a first nonlinear layer of an encoder subnetwork. Process 700 may include processing output values associated with at least one chrominance channel of a frame using a second nonlinear layer of the encoder sub-network. In such instances, the combined representation is generated based on the output of the first nonlinear layer and the output of the second nonlinear layer. In some cases, the combined representation of the frame is formed by a third convolutional layer (e.g., the 1x1 convolutional layer of the encoder subnetwork of Figures 6A-6F ) using the output of the first nonlinear layer and the output of the second nonlinear layer. output is generated as input.

In some examples, process 700 includes quantizing the encoded video data (eg, using quantization engine 620). In some examples, process 700 includes entropy coding the encoded video data (eg, using entropy coding engine 621 ). In some examples, process 700 includes storing the encoded video data in memory. In some examples, process 700 includes sending the encoded video data to at least one device over a transmission medium.

In some examples, process 700 includes obtaining an encoded frame. Process 700 may include generating, by a first convolutional layer of a decoder sub-network of the neural network system, a reconstructed output value associated with a luma channel of the encoded frame. Process 700 may further include generating, by a second convolutional layer of the decoder subnetwork, a reconstructed output value associated with at least one chrominance channel of the encoded frame. In some examples, process 700 includes using a third convolutional layer of the decoder subnetwork to separate a luma channel of the encoded frame from at least one chrominance channel of the encoded frame. In some cases, the third convolutional layer of the decoder subnetwork includes a 1x1 convolutional layer (eg, the 1x1 convolutional layer of the decoder subnetwork of FIGS. 6A-6F ) that includes one or more 1x1 convolutional filters.

In some examples, the frames include video frames. In some examples, the at least one chroma channel includes a chroma blue channel and a chroma red channel. In some examples, the frame has a luma-chroma (YUV) format.

8 is a flowchart illustrating an example of a process 800 for processing video using one or more machine learning techniques described herein. At block 802, the process 800 includes obtaining an encoded frame. At block 804, the process 800 includes separating, by a first convolutional layer of the decoder subnetwork, a luma channel of the encoded frame from at least one chrominance channel of the encoded frame. In some cases, the first convolutional layer of the decoder subnetwork includes a 1x1 convolutional layer (eg, the 1x1 convolutional layer of the decoder subnetwork of FIGS. 6A-6F ) that includes one or more 1x1 convolutional filters. At block 806, the process 800 includes generating, by a second convolutional layer of the decoder sub-network of the neural network system, a reconstructed output value associated with a luma channel of the encoded frame. At block 808, the process 800 includes generating, by a third convolutional layer of the decoder subnetwork, a reconstructed output value associated with at least one chrominance channel of the encoded frame. At block 810, the process 800 includes generating an output frame including reconstructed output values associated with a luma channel and reconstructed output values associated with at least one chroma channel.

In some examples, process 800 includes processing values associated with a luma channel of the encoded frame using a first nonlinear layer of the decoder subnetwork. A reconstructed output value associated with the luma channel is generated based on the output of the first nonlinear layer. Process 700 can include processing values associated with at least one chrominance channel of the encoded frame using a second nonlinear layer of the decoder subnetwork. A reconstructed output value associated with at least one chroma channel is generated based on the output of the second nonlinear layer.

In some examples, process 800 includes dequantizing (eg, via dequantization engine 623 ) samples of the encoded frame. In some examples, process 800 includes entropy decoding (eg, via entropy decoding engine 622 ) samples of the encoded frame. In some examples, process 800 includes storing the output frame in memory. In some examples, process 800 includes displaying an output frame.

In some examples, process 800 includes generating, by a first convolutional layer of an encoder sub-network of the neural network system, an output value associated with a luma channel of the frame. Process 700 may include generating, by a second convolutional layer of the encoder sub-network, an output value associated with at least one chrominance channel of the frame. Process 700 may further include generating, by a third convolutional layer of the encoder sub-network, a combination of frames based on an output value associated with a luma channel of the frame and an output value associated with at least one chrominance channel of the frame express. Process 700 may include generating an encoded frame based on the combined representation of the frame. In some cases, the third convolutional layer of the encoder subnetwork includes a 1x1 convolutional layer (eg, the 1x1 convolutional layer of the encoder subnetwork of FIGS. 6A-6F ) that includes one or more 1x1 convolutional filters.

In some examples, process 800 includes processing an output value associated with a luma channel of a frame using a first nonlinear layer of an encoder subnetwork, and processing an output value associated with a luma channel of a frame using a second nonlinear layer of an encoder subnetwork. An output value associated with at least one chroma channel of the frame. In such instances, the combined representation is generated based on the output of the first nonlinear layer and the output of the second nonlinear layer. In some examples, the combined representation of the frame is produced by a third convolutional layer of the encoder sub-network using the output of the first nonlinear layer and the output of the second nonlinear layer as input.

In some examples, the encoded frames include encoded video frames. In some examples, the at least one chroma channel includes a chroma blue channel and a chroma red channel. In some examples, the encoded frames have a luma-chroma (YUV) format.

In some examples, the programs described herein (e.g., program 700 described herein, program 800, and/or other programs described herein) can be executed by a computing device or apparatus, such as a computer having a computing device architecture 900 shown in FIG. computing device) to perform. In one example, the program 700 and/or the program 800 can be executed by a computing device, wherein the computing device architecture 900 implements one of the neural network architectures shown in FIGS. 6A-6F . In some examples, computing devices may include mobile devices (e.g., mobile phones, tablet computing devices), wearable devices, extended reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality ( MR) equipment), personal computers, laptops, video servers, televisions, vehicles (or computing equipment for vehicles), robotic equipment and/or have a program for executing the programs described herein (including program 700 and/or program 800) resource capabilities of any other computing device.

In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers , one or more transmitters, receivers or combination transmitter-receivers (for example, referred to as transceivers), one or more cameras, one or more sensors, and/or configured to perform the Other components of the procedure's steps. In some examples, a computing device may include a display, a network interface configured to transmit and/or receive data, any combination thereof, and/or other components. A network interface may be configured to transmit and/or receive Internet Protocol (IP)-based data or other types of data.

Components of a computing device may be implemented in circuits. For example, a component may include and/or be implemented using electronic circuits or other electronic hardware that may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPU), digital signal processor (DSP), central processing unit (CPU), neural processing unit (NPU), and/or other appropriate electronic circuits), and/or may include computer software, firmware, or any combination thereof and/or implemented using same to perform various operations described herein.

Program 700 and program 800 are shown as logic flow diagrams, the actions of which represent a series of operations that can be implemented by hardware, computer instructions, or a combination thereof. In the context of computer instructions, the actions mean computer-executable instructions stored on one or more computer-readable storage media which, when executed by one or more processors, perform the described operate. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and/or in parallel to implement the procedure.

Additionally, the programs described herein (including program 700, program 800, and/or other programs described herein) can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as or codes (for example, executable instructions, one or more computer programs, or one or more applications) that are jointly executed on multiple processors, implemented through hardware, or a combination thereof. As mentioned above, code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. A computer-readable storage medium or a machine-readable storage medium may be non-transitory.

FIG. 9 illustrates an example computing device architecture 900 for an example computing device that may implement various techniques described herein. In some examples, computing devices may include mobile devices, wearable devices, extended reality devices (eg, virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), personal computers, laptop computer, video server, vehicle (or a vehicle's computing device), or other equipment. For example, computing device architecture 900 may implement the system of FIG. 6 . Components of computing device architecture 900 are shown as being in electrical communication with each other using connections 905 (eg, bus bars). Example computing device architecture 900 includes processing unit (CPU or processor) 910 and computing device connection 905, which will include computing device memory 915, such as read-only memory (ROM) 920 and random access memory ( Various computing device components such as RAM) 925 ) are coupled to the processor 910 .

Computing device architecture 900 may include high-speed memory cache memory coupled directly to processor 910 , proximate to processor 910 , or integrated as part of processor 910 . The computing device architecture 900 can copy data from the memory 915 and/or the storage device 930 to the cache memory 912 for fast access by the processor 910 . In this way, cache memory can provide a performance boost that prevents the processor 910 from being delayed while waiting for data. These and other modules may control or be configured to control the processor 910 to perform various actions. Other computing device memory 915 may also be available. Memory 915 may include a variety of different types of memory with different performance characteristics. Processor 910 may include any general-purpose processor and hardware or software services (such as Service 1 932, Service 2 934, and Service 3 936 stored in storage device 930) configured to control processor 910 and special purpose processors in which software instructions are incorporated into the processor design. Processor 910 may be a self-contained system that includes multiple cores or processors, a bus, memory controller, cache memory, and the like. Multi-core processors can be symmetric or asymmetric.

To enable user interaction with computing device architecture 900, input devices 945 may represent any number of input mechanisms, such as a microphone for voice, touch screen for gesture or graphical input, keyboard, mouse, motion input, voice etc. The output device 935 may also be one or more of various output mechanisms known to those skilled in the art, such as a display, projector, television, speaker device, and the like. In some cases, a multimodal computing device may enable a user to provide multiple types of input to communicate with computing device architecture 900 . Communication interface 940 can generally control and manage user input and computing device output. There is no restriction on operation on any particular hardware setup, and as such, the basic features herein can easily be replaced by improved hardware or firmware configurations as they are developed.

Storage device 930 is non-volatile memory and can be a hard disk or other type of computer-readable media that can store data that can be accessed by the computer, such as magnetic tape, flash memory cards, solid-state memory devices, digital versatile magnetic Discs, cassette tapes, random access memory (RAM) 925, read only memory (ROM) 920, and mixtures thereof. The repository 930 may include a service 932 , a service 934 , and a service 936 for controlling the processor 910 . Additional hardware or software mods are expected. Storage device 930 may be connected to computing device connection 905 . In one aspect, a hardware module that performs a specific function may include software components stored in a computer-readable medium that are connected to hardware components necessary to perform the function, such as the processor 910, 905, output device 935, etc.) are connected.

Aspects of this disclosure apply to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other equipment). Although described below with respect to a device having or being coupled to one light projector, aspects of the disclosure apply to devices having any number of light projectors, and thus are not limited to a particular device.

The term "device" is not limited to one or a specific number of physical objects (such as a smartphone, a controller, a processing system, etc.). As used herein, a device may be any electronic device having one or more components that can implement at least some portions of this disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a particular configuration, type, or number of items. Additionally, the term "system" is not limited to a number of components or a specific embodiment. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have moving or static parts. Although the following description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a particular configuration, type, or number of items.

In the above description specific details are provided to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those having ordinary skill in the art to which the present invention pertains that these embodiments may be practiced without these specific details. For clarity of explanation, in some cases the technology herein may be presented as comprising individual functional blocks comprising functional blocks comprising a device, a component of a device, a step or routine in a method embodied in software, or A combination of hardware and software. Additional components may be used in addition to those shown in the figures and/or described herein. For example, circuits, systems, networks, programs and other components may be shown in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, procedures, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

The various embodiments above may be described as procedures or methods, and the procedures or methods are illustrated as flowcharts, schematic flowcharts, data flow diagrams, structural diagrams or block diagrams. Although a flowchart may describe operations as a sequential program, many of these operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A program is terminated when its operations are complete, but may have additional steps not included in the figure. A procedure may correspond to a method, a function, a procedure, a subroutine, a subroutine, and the like. When a program corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

Programs and methods according to the above examples can be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions or material which cause a general purpose computer, special purpose computer or processing device to execute or otherwise configure it to perform a particular function or a particular set of functions. The portion of computer resources used that can be accessed via the Internet. Computer-executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, and the like.

The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices and various other media capable of storing, containing or carrying instructions and/or data. Computer-readable media may include non-transitory media in which data can be stored and do not include carrier waves and/or transitory electronic signals that travel wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as flash memory, memory or memory devices, magnetic disks or optical disks, flash memory, provided with non-volatile memory USB devices, network storage devices, compact discs (CDs) or digital versatile discs (DVDs), any suitable combination thereof, etc. The computer-readable medium may have code and/or machine-executable instructions stored thereon, which may represent a program, function, subroutine, program, routine, subroutine, module, package, software component, or Any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means including memory sharing, message passing, token passing, network transmission, and the like.

In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals including bit streams and the like. However, when mentioned, non-transitory computer readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

An apparatus implementing programs and methods in accordance with these disclosures may comprise hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof and may take any of a variety of form factors. When implemented in software, firmware, middleware or microcode, the code or code segments (eg, a computer program product) to perform the necessary tasks may be stored on a computer-readable or machine-readable medium. The processor can perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack mountable devices, standalone devices, and the like. The functions described herein may also be embodied in peripheral devices or add-on cards. By way of additional example, such functionality may also be implemented on circuit boards between different chips or different processes executing in a single device.

Instructions, the media for carrying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example elements for providing the functionality described in this disclosure.

In the foregoing description, aspects of the invention have been described with reference to particular embodiments of the invention, but those of ordinary skill in the art to which the invention pertains will recognize that the invention is not limited thereto. Therefore, while illustrative embodiments of the present case have been described in detail herein, it is to be understood that the inventive concept may be variously embodied and employed, and the appended claims are intended to be construed to include such variations, except Variants limited by prior art. The various features and aspects of the applications described above can be used individually or in combination. Furthermore, the embodiments may be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. For purposes of illustration, the methods are described in a particular order. It should be appreciated that, in alternative embodiments, the methods may be performed in an order different from that described.

Those of ordinary skill in the art to which the present invention pertains will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or greater than, respectively, without departing from the scope of the present description. equal to ("≦") and greater than or equal to ("≧") symbols.

Where a component is described as being "configured" to perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform that operation, by programming circuits (e.g., microprocessor or other suitable circuitry) programmed to perform that operation, or any combination thereof.

The phrase "coupled to" means any component that is directly or indirectly physically connected to another component and/or any component that directly or indirectly communicates with another component (for example, via a wired or wireless connection and/or other suitable connected to another component via its communication interface).

Claim language or other language stating "at least one" of a set and/or "one or more" of a set indicates that a member of the set or members of the set satisfy the claim. For example, claim language that states "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. "At least one" of a language set and/or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally be included in the set of A and B Items not listed.

The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those with ordinary knowledge may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present case.

The techniques described herein may also be implemented with electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices that have multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as separate but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium that includes program code that, when executed, performs one or more of the methods described above. Directives for multiple methods. A computer readable data storage medium may form a part of a computer program product, which may include packaging materials. Computer readable media may include memory or data storage media such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile RAM Access memory (NVRAM), electronically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, such techniques may be implemented at least in part by computer-readable communication media (such as propagated signal or wave) to achieve.

The code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable Logic array (FPGA) or other equivalent integrated or individual logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor can be a microprocessor, but in the alternative, the processor can be any general processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Accordingly, the term "processor," as used herein may represent any of the foregoing structure, any combination of the foregoing, or any other structure or device suitable for implementation of the techniques described herein.

Illustrative examples of the content of this case include:

Aspect 1: A method of processing video data, the method comprising: generating an output value associated with a luma channel of a frame by a first convolutional layer of an encoder subnetwork of a neural network system; The second convolutional layer of the way generates output values associated with at least one chrominance channel of the frame; the output value associated with the luma channel of the frame is generated by the third convolutional layer based on the output value associated with the luma channel of the frame and the at least one generating a combined representation of the frame from an associated output value of a chrominance channel; and generating encoded video data based on the combined representation of the frame.

Aspect 2: The method of Aspect 1, wherein the third convolutional layer includes a 1x1 convolutional layer, and the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

Aspect 3: The method of any one of Aspects 1 or 2, further comprising: using the first non-linear layer of the encoder sub-network to process the an output value; and processing the output value associated with the at least one chrominance channel of the frame using a second nonlinear layer of the encoder subnetwork; wherein the combined representation is based on the output of the first nonlinear layer and the output of this second nonlinear layer is generated.

Aspect 4: The method according to Aspect 3, wherein the combined representation of the frame is generated by the third convolutional layer using the output of the first nonlinear layer and the output of the second nonlinear layer as input.

Aspect 5: The method according to any one of Aspects 1 to 4, further comprising: quantizing the encoded video data.

Aspect 6: The method according to any one of aspects 1 to 5, further comprising: performing entropy decoding on the decoded video data.

Aspect 7: The method according to any one of Aspects 1 to 6, further comprising: storing the encoded video data in a memory.

Aspect 8: The method according to any one of aspects 1 to 7, further comprising: sending the encoded video data to at least one device over a transmission medium.

Aspect 9: The method according to any one of aspects 1 to 8, further comprising: obtaining an encoded frame; generated and encoded by the first convolutional layer of the decoder sub-network of the neural network system the reconstructed output value associated with the luma channel of the frame; and the reconstructed output associated with at least one chrominance channel of the encoded frame produced by the second convolutional layer of the decoder subnetwork value.

Aspect 10: The method of Aspect 9, further comprising: using a third convolutional layer of the decoder subnetwork to combine the luma channel of the encoded frame with the at least one chrominance channel of the encoded frame separate.

Aspect 11: The method of Aspect 10, wherein the third convolutional layer of the decoder subnetwork includes a 1x1 convolutional layer, the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

Aspect 12: The method of any one of Aspects 1 to 11, wherein the frame comprises a video frame.

Aspect 13: The method of any one of Aspects 1 to 12, wherein the at least one chroma channel includes a chroma blue channel and a chroma red channel.

Aspect 14: The method of any one of Aspects 1 to 13, wherein the frame has a luma-chroma (YUV) format.

Aspect 15: A device for processing video data. The apparatus includes: a memory; and a processor coupled to the memory and configured to: generate an output value associated with a luma channel of a frame using a first convolutional layer of an encoder sub-network of the neural network system ; using the second convolutional layer of the encoder subnetwork to generate an output value associated with at least one chrominance channel of the frame; using a third convolutional layer based on the output value associated with the luma channel of the frame and output values associated with the at least one chrominance channel of the frame to generate a combined representation of the frame; and generating encoded video data based on the combined representation of the frame.

Aspect 16: The device of Aspect 15, wherein the third convolutional layer comprises a 1x1 convolutional layer comprising one or more 1x1 convolutional filters.

Aspect 17: The apparatus of any of Aspects 15 or 16, wherein the processor is configured to process the luma channel of the frame using a first non-linear layer of the encoder subnetwork the associated output values; and process the output values associated with the at least one chrominance channel of the frame using a second nonlinear layer of the encoder sub-network; wherein the combined representation is based on the first The output of the nonlinear layer and the output of the second nonlinear layer are generated.

Aspect 18: The apparatus according to Aspect 17, wherein the combined representation of the frame is generated by the third convolutional layer using the output of the first nonlinear layer and the output of the second nonlinear layer as input.

Aspect 19: The apparatus of any of Aspects 15-18, wherein the processor is configured to: quantize the encoded video data.

Aspect 20: The apparatus of any of Aspects 15-19, wherein the processor is configured to entropy decode the encoded video data.

Aspect 21: The apparatus of any of Aspects 15-20, wherein the processor is configured to: store the encoded video data in memory.

Aspect 22: The apparatus of any one of Aspects 15 to 21, wherein the processor is configured to send the encoded video data to at least one device over a transmission medium.

Aspect 23: The apparatus of any one of Aspects 15 to 22, wherein the processor is configured to: obtain an encoded frame; use a first convolution of the decoder subnetwork of the neural network system layer to generate a reconstructed output value associated with the luma channel of the encoded frame; and use the second convolutional layer of the decoder subnetwork to generate Linked reconstructed output value.

Aspect 24: The device of Aspect 23, wherein the processor is configured to use a third convolutional layer of the decoder subnetwork to combine the luma channel of the encoded frame with the At least one chroma channel is separated.

Aspect 25: The device of Aspect 24, wherein the third convolutional layer of the decoder subnetwork includes a 1x1 convolutional layer, the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

Aspect 26: The device of any one of Aspects 15-25, wherein the frame comprises a video frame.

Aspect 27: The device of any of Aspects 15-26, wherein the at least one chroma channel includes a chroma blue channel and a chroma red channel.

Aspect 28: The device of any one of Aspects 15-27, wherein the frame has a luma-chroma (YUV) format.

Aspect 29: The apparatus of any one of Aspects 15 to 28, wherein the processor comprises a Neural Processing Unit (NPU).

Aspect 30: The apparatus of any one of Aspects 15-29, wherein the apparatus comprises a mobile device.

Aspect 31: The apparatus of any of Aspects 15-30, wherein the apparatus comprises an extended reality device.

Aspect 32: The device according to any one of Aspects 15 to 31, further comprising a display.

Aspect 33: The device of any one of Aspects 15 to 29, wherein the device comprises a television.

Aspect 34: The device of any of Aspects 15-33, wherein the device comprises a camera configured to capture one or more video frames.

Aspect 35: A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform any one of the operations of Aspects 1-14.

Aspect 36: An apparatus comprising means for performing any one of the operations according to Aspects 1-14.

Aspect 37: A method of processing video data, the method comprising: obtaining an encoded frame; combining, by a first convolutional layer of the decoder subnetwork, a luminance channel of the encoded frame with the encoded frame separation of at least one chrominance channel of the frame; a second convolutional layer of the decoder subnetwork of the neural network system produces a reconstructed output value associated with the luma channel of the encoded frame; by the decoding The third convolutional layer of the device subnetwork generates the reconstructed output value associated with the at least one chrominance channel of the encoded frame; and generates an output frame including The associated reconstructed output value and the reconstructed output value associated with the at least one chroma channel.

Aspect 38: The method of Aspect 37, wherein the first convolutional layer of the decoder subnetwork includes a 1x1 convolutional layer, the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

Aspect 39: The method of any one of Aspects 37 or 38, further comprising: using the first non-linear layer of the decoder subnetwork to process the luminance channel associated with the encoded frame values, wherein the reconstructed output values associated with the luma channel are generated based on the output of the first nonlinear layer; and using the second nonlinear layer of the decoder subnetwork to process the Values associated with the at least one chroma channel of the encoded frame, wherein the reconstructed output values associated with the at least one chroma channel are generated based on the output of the second nonlinear layer.

Aspect 40: The method of any one of aspects 37-39, further comprising: dequantizing the samples of the encoded frame.

Aspect 41: The method of any of Aspects 37-40, further comprising: entropy decoding the samples of the encoded frame.

Aspect 42: The method according to any one of Aspects 37 to 41, further comprising: storing the output frame in a memory.

Aspect 43: The method according to any one of Aspects 37 to 42, further comprising: displaying the output frame.

Aspect 44: The method of any one of Aspects 37 to 43, further comprising: generating, by the first convolutional layer of the encoder sub-network of the neural network system, an output associated with a luma channel of a frame value; an output value associated with at least one chrominance channel of the frame is generated by the second convolutional layer of the encoder subnetwork; an output value associated with at least one chrominance channel of the frame is generated by the third convolutional layer of the encoder subnetwork based on the an output value associated with a luma channel and an output value associated with the at least one chrominance channel of the frame to generate a combined representation of the frame; and generating the encoded frame based on the combined representation of the frame .

Aspect 45: The method of Aspect 44, wherein the third convolutional layer of the encoder sub-network includes a 1x1 convolutional layer, the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

Aspect 46: The method of any one of Aspects 44 or 45, further comprising: using the first nonlinear layer of the encoder subnetwork to process output values associated with the luma channel of the frame; and using a second nonlinear layer of the encoder subnetwork to process output values associated with the at least one chroma channel of the frame; wherein the combined representation is based on the output of the first nonlinear layer and the second The output of the second nonlinear layer is generated.

Aspect 47: The method according to aspect 46, wherein the combined representation of the frame is obtained by the third convolutional layer of the encoder subnetwork using the output of the first nonlinear layer and the output of the second nonlinear layer output is generated as input.

Aspect 48: The method of any of Aspects 37-47, wherein the encoded frame comprises an encoded video frame.

Aspect 49: The method of any one of Aspects 37 to 48, wherein the at least one chroma channel includes a chroma blue channel and a chroma red channel.

Aspect 50: The method of any of Aspects 37-49, wherein the encoded frame has a luma-chrominance (YUV) format.

Aspect 49: A device for processing video data. The device includes: a memory; and a processor coupled to the memory and configured to: obtain an encoded frame; use the first convolutional layer of the decoder subnetwork to a luma channel is separated from at least one chrominance channel of the encoded frame; using a second convolutional layer of a decoder sub-network of the neural network system to generate a chroma channel associated with the luma channel of the encoded frame a reconstructed output value; using a third convolutional layer of the decoder subnetwork to generate a reconstructed output value associated with the at least one chrominance channel of the encoded frame; and generating an output frame, The output frame includes reconstructed output values associated with the luma channel and reconstructed output values associated with the at least one chroma channel.

Aspect 50: The device of Aspect 49, wherein the first convolutional layer of the decoder subnetwork includes a 1x1 convolutional layer, the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

Aspect 51: The apparatus of any one of Aspects 49 or 50, wherein the processor is configured to use the first non-linear layer of the decoder subnetwork to process the encoded frame a value associated with a luma channel, wherein the reconstructed output values associated with the luma channel are generated based on the output of the first nonlinear layer; and using a second nonlinearity of the decoder sub-network layer to process values associated with the at least one chroma channel of the encoded frame, wherein the reconstructed output values associated with the at least one chroma channel are based on the second nonlinear layer output is generated.

Aspect 52: The apparatus of any one of Aspects 49-51, wherein the processor is configured to dequantize samples of the encoded frame.

Aspect 53: The apparatus of any one of Aspects 49 to 52, wherein the processor is configured to entropy decode the samples of the encoded frame.

Aspect 54: The device of any one of Aspects 49 to 53, wherein the processor is configured to: store the output frame in memory.

Aspect 55: The device of any one of Aspects 49 to 54, wherein the processor is configured to: display the output frame.

Aspect 56: The apparatus of any one of Aspects 49 to 55, wherein the processor is configured to generate a frame-intensity correlation with the first convolutional layer of the encoder sub-network of the neural network system an output value associated with a channel; an output value associated with at least one chrominance channel of the frame is generated by the second convolutional layer of the encoder subnetwork; an output value associated with at least one chrominance channel of the frame is generated by a third convolutional layer of the encoder subnetwork based on the an output value associated with the luma channel of the frame and an output value associated with the at least one chrominance channel of the frame to generate a combined representation of the frame; and generating the combined representation of the frame based on the combined representation of the frame Encoded frame.

Aspect 57: The device of Aspect 56, wherein the third convolutional layer of the encoder sub-network includes a 1x1 convolutional layer, the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

Aspect 58: The apparatus of any one of Aspects 44 or 57, wherein the processor is configured to use the first nonlinear layer of the encoder sub-network to process and using a second nonlinear layer of the encoder subnetwork to process output values associated with the at least one chrominance channel of the frame; wherein the combined representation is based on the first nonlinear layer The output of and the output of this second nonlinear layer are produced.

Aspect 59: The apparatus according to Aspect 58, wherein the combined representation of the frame is obtained by the third convolutional layer of the encoder subnetwork using the output of the first nonlinear layer and the output of the second nonlinear layer output is generated as input.

Aspect 60: The device of any one of Aspects 49-59, wherein the encoded frame comprises an encoded video frame.

Aspect 61: The device of any one of Aspects 49 to 60, wherein the at least one chroma channel includes a chroma blue channel and a chroma red channel.

Aspect 62: The device of any one of Aspects 49-61, wherein the encoded frame has a luma-chrominance (YUV) format.

Aspect 63: The apparatus of any one of Aspects 49 to 62, wherein the processor comprises a Neural Processing Unit (NPU).

Aspect 64: The apparatus of any one of Aspects 49 to 63, wherein the apparatus comprises a mobile device.

Aspect 65: The apparatus of any one of Aspects 49 to 64, wherein the apparatus comprises an extended reality device.

Aspect 66: The device of any one of Aspects 49 to 65, further comprising a display.

Aspect 67: The device of any one of Aspects 49 to 63, wherein the device comprises a television.

Aspect 68: The device of any of Aspects 49-67, wherein the device comprises a camera configured to capture one or more video frames.

Aspect 69: A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform any of the operations according to Aspects 37-48.

Aspect 70: An apparatus comprising means for performing any of the operations according to aspects 37-48.

100:SOC 102:CPU 104:GPU 106:DSP 108:Neural Processing Unit (NPU) 110:Connection Block 112:Multimedia Processor 114:Sensor Processor 116:Image Signal Processor (ISP) 118:Memory Block 120: navigation module 202: fully connected neural network 204: fully connected neural network 206: convolutional neural network 210: value 212: value 214: value 216: value 218: first set 220: second set 222: output 224: first feature vector 226: image 228: second feature vector 230: image capture device 232: convolution layer 350: depth convolution network 352: input data 354A: convolution block 354B: convolution block 356: layer 358: layer 360 : layer 362A: layer 362B: layer 364: layer 366: classification score 400: system 402: equipment 404: processor 406: memory 407: camera 408: image data 410: E2E-NNVC system 412: first interface ("I /F 1") 414: storage medium 416: second interface ("I/F 2") 418: transmission medium 462: encoder section 463: neural network 464: quantizer 466: decoder section 468: neural network 470: Input Data 472: Intermediate Data 474: Output Data 476: Representation 490: Second Device 510: First Convolution Layer 602: Luma (Y) Channel 604: Chroma (U and V) Channel 606: 1x1 Convolution Layer 608: GDN nonlinear operator 609: inverse GDN (IGDN) operator 610: second convolution layer 611: convolution layer 612: second GDN layer 613: 1x1 convolution layer 614: first GDN layer 615: IGDN 616: IGDN 617: convolution Layer 618: Convolution Layer 619: Final Neural Network Layer 620: Quantization Engine 621: Entropy Encoding Engine 622: Entropy Decoding Engine 623: Dequantization Engine 624: Y Component 625: U and V Component 632: N Channel Chroma Output 634: N-channel brightness output 638: 1x1 convolution layer 639: output value 642: brightness channel 644: chroma channel 646: convolution layer 647: convolution layer 648: 1x1 convolution layer 649: pReLU nonlinear operation unit 652: pReLU layer 654: pReLU layer 662: pReLU layer 664: pReLU layer 670: convolution layer 671: convolution layer 700: program 702: block 704: block 706: block 708: block 800: program 802: block 804: block 806: block 808: block 810: block 900 : Computing Device Architecture 905: Connection 910: Processor 912: Cache Memory 915: Computing Device Memory 920: Read Only Memory (ROM) 925: Random Access Memory (RAM) 930: Storage Device 932: Server Service 934: service 935: output device 936: service 940: communication interface 945: input device C1: first channel C2: second channel g _a : subnet g _s : subnet GDN: normalization IGDN: reverse normalization pReLU: Parametric Correction Linear Unit S ₁ : Value S ₂ : Value U: Chroma V: Chroma Y: Brightness

Illustrative embodiments of the present case are described in detail below with reference to the following drawings:

Figure 1 shows an example implementation of a system on chip (SOC);

Figure 2A shows an example of a fully connected neural network;

Figure 2B shows an example of a locally connected neural network;

Figure 2C shows an example of a convolutional neural network;

Figure 2D shows a detailed example of a deep convolutional network (DCN) designed to recognize visual features from images;

Figure 3 is a block diagram illustrating a deep convolutional network (DCN);

4 is a schematic diagram illustrating an example of a system including an apparatus operable to perform image and/or video coding (encoding and decoding) using a neural network-based system, according to some examples;

5 is a schematic diagram illustrating an example of an end-to-end neural network-based image and video coding system for an input having a red-green-blue (RGB) format, according to some examples;

6A is a schematic diagram illustrating an example of a front-end neural network architecture that may be part of an end-to-end neural network-based image and video decoding system, according to some examples;

6B is a schematic diagram illustrating example operation of a 1x1 convolutional layer according to some examples;

6C is a schematic diagram illustrating another example of a front-end neural network architecture, which may be part of an end-to-end neural network-based image and video coding system, according to some examples;

6D is a schematic diagram illustrating another example of a front-end neural network architecture that may be part of an end-to-end neural network-based image and video coding system, according to some examples;

6E is a schematic diagram illustrating another example of a front-end neural network architecture that may be part of an end-to-end neural network-based image and video coding system, according to some examples;

6F is a schematic diagram illustrating another example of a front-end neural network architecture that may be part of an end-to-end neural network-based image and video coding system, according to some examples;

7 is a flowchart illustrating an example of a program for processing video data, according to some examples;

8 is a flowchart illustrating another example of a program for processing video data, according to some examples; and

9 illustrates an example computing device architecture of an example computing device that can implement various techniques described herein.

Domestic deposit information (please note in order of depositor, date, and number) none Overseas storage information (please note in order of storage country, institution, date, and number) none

632: N channel chroma output

634:N channel brightness output

638: 1x1 Convolution Layer

639: output value

Claims (60)

A method for processing video data, the method includes the following steps: generating output values associated with a luma channel of a frame by a first convolutional layer of an encoder subnetwork of a neural network system; generating output values associated with at least one chrominance channel of the frame from a second convolutional layer of the encoder subnetwork; generating a combined representation of the frame by a third convolutional layer based on the output values associated with the luma channel of the frame and the output values associated with the at least one chrominance channel of the frame ;and Encoded video data is generated based on the combined representation of the frame. The method according to claim 1, wherein the third convolutional layer includes a 1x1 convolutional layer, and the 1x1 convolutional layer includes one or more 1x1 convolutional filters. The method according to Claim 1 also includes the following steps: processing the output values associated with the luma channel of the frame using a first non-linear layer of the encoder subnetwork; and processing output values associated with the at least one chrominance channel of the frame using a second nonlinear layer of the encoder subnetwork; Wherein the combined representation is generated based on an output of the first nonlinear layer and an output of the second nonlinear layer. The method according to claim 3, wherein the combined representation of the frame is generated by the third convolutional layer using an output of the first nonlinear layer and an output of the second nonlinear layer as input. The method according to Claim 1 also includes the following steps: The encoded video data is quantized. The method according to Claim 1 also includes the following steps: Entropy coding is performed on the encoded video data. The method according to Claim 1 also includes the following steps: The encoded video data is stored in memory. The method according to Claim 1 also includes the following steps: The encoded video data is sent to at least one device over a transmission medium. The method according to Claim 1 also includes the following steps: obtain an encoded frame; generating reconstructed output values associated with a luma channel of the encoded frame from a first convolutional layer of a decoder subnetwork of the neural network system; and Reconstructed output values associated with at least one chrominance channel of the encoded frame are generated by a second convolutional layer of the decoder subnetwork. The method according to claim 9 also includes the following steps: A third convolutional layer of the decoder subnetwork is used to separate the luma channel of the encoded frame from the at least one chrominance channel of the encoded frame. The method according to claim 10, wherein the third convolutional layer of the decoder sub-network includes a 1x1 convolutional layer, the 1x1 convolutional layer includes one or more 1x1 convolutional filters. The method according to claim 1, wherein the frame includes a video frame. The method according to claim 1, wherein the at least one chroma channel includes a chroma blue channel and a chroma red channel. The method according to claim 1, wherein the frame has a luma-chroma (YUV) format. A device for processing video data, comprising: a memory; and A processor, coupled to the memory and configured to: using a first convolutional layer of an encoder sub-network of a neural network system to generate output values associated with a luma channel of the frame; generating output values associated with at least one chrominance channel of the frame using a second convolutional layer of the encoder subnetwork; using a third convolutional layer to generate a combined representation of the frame based on output values associated with the luma channel of the frame and output values associated with the at least one chrominance channel of the frame; and Encoded video data is generated based on the combined representation of the frame. The device according to claim 15, wherein the third convolutional layer comprises a 1x1 convolutional layer, and the 1x1 convolutional layer comprises one or more 1x1 convolutional filters. The device according to claim 15, wherein the processor is configured to: processing the output values associated with the luma channel of the frame using a first non-linear layer of the encoder subnetwork; and processing output values associated with the at least one chrominance channel of the frame using a second nonlinear layer of the encoder subnetwork; Wherein the combined representation is generated based on an output of the first nonlinear layer and an output of the second nonlinear layer. The apparatus according to claim 17, wherein the combined representation of the frame is generated by the third convolutional layer using an output of the first nonlinear layer and an output of the second nonlinear layer as input. The device according to claim 15, wherein the processor is configured to: The encoded video data is quantized. The device according to claim 15, wherein the processor is configured to: Entropy encoding is performed on the encoded video data. The apparatus according to any one of claim 15, wherein the processor is configured to: The encoded video data is stored in memory. The device according to claim 15, wherein the processor is configured to: The encoded video data is sent to at least one device over a transmission medium. The device according to claim 15, wherein the processor is configured to: obtain an encoded frame; using a first convolutional layer of a decoder subnetwork of the neural network system to generate reconstructed output values associated with a luma channel of the encoded frame; and A second convolutional layer of the decoder subnetwork is used to generate reconstructed output values associated with at least one chrominance channel of the encoded frame. The device according to claim 23, wherein the processor is configured to: A third convolutional layer of the decoder subnetwork is used to separate the luma channel of the encoded frame from the at least one chrominance channel of the encoded frame. The apparatus according to claim 24, wherein the third convolutional layer of the decoder sub-network comprises a 1x1 convolutional layer, the 1x1 convolutional layer comprises one or more 1x1 convolutional filters. The device according to claim 15, wherein the frame includes a video frame. The device according to claim 15, wherein the at least one chroma channel comprises a chroma blue channel and a chroma red channel. The device according to claim 15, wherein the frame has a luma-chroma (YUV) format. The device according to claim 15, wherein the processor comprises a Neural Processing Unit (NPU). The device according to claim 15, wherein the device comprises a mobile device. The device according to claim 15, further comprising: at least one of a display and a camera configured to capture one or more frames. A method for processing video data, the method includes the following steps: obtain an encoded frame; separating a luma channel of the encoded frame from at least one chrominance channel of the encoded frame by a first convolutional layer of a decoder subnetwork; generating, by a second convolutional layer of the decoder subnetwork of a neural network system, reconstructed output values associated with the luma channel of the encoded frame; generating reconstructed output values associated with the at least one chrominance channel of the encoded frame from a third convolutional layer of the decoder subnetwork; and An output frame is generated, the output frame including the reconstructed output values associated with the luma channel and the reconstructed output values associated with the at least one chroma channel. The method according to claim 32, wherein the first convolutional layer of the decoder sub-network includes a 1x1 convolutional layer, the 1x1 convolutional layer includes one or more 1x1 convolutional filters. The method according to claim 32 also includes the following steps: using a first non-linear layer of the decoder subnetwork to process values associated with the luma channel of the encoded frame, wherein the reconstructed output values associated with the luma channel are based on produced by an output of the first nonlinear layer; and Values associated with the at least one chroma channel of the encoded frame are processed using a second non-linear layer of the decoder subnetwork, wherein the weighted values associated with the at least one chroma channel The structured output value is generated based on an output of the second nonlinear layer. The method according to claim 32 also includes the following steps: The samples of the encoded frame are dequantized. The method according to claim 32 also includes the following steps: The samples of the encoded frame are entropy decoded. The method according to claim 32 also includes the following steps: Store the output frame in memory. The method according to claim 32 also includes the following steps: Display the output frame. The method according to claim 32 also includes the following steps: generating an output value associated with a luma channel of the frame from a first convolutional layer of an encoder sub-network of the neural network system; generating output values associated with at least one chrominance channel of the frame from a second convolutional layer of the encoder subnetwork; generating the frame by a third convolutional layer of the encoder subnetwork based on output values associated with the luma channel of the frame and output values associated with the at least one chrominance channel of the frame a combined representation; and The encoded frame is generated based on the combined representation of the frame. The method according to claim 39, wherein the third convolutional layer of the encoder sub-network includes a 1x1 convolutional layer, the 1x1 convolutional layer includes one or more 1x1 convolutional filters. The method according to claim 39 also includes the following steps: using a first nonlinear layer of the encoder subnetwork to process output values associated with the luma channel of the frame; and processing output values associated with the at least one chrominance channel of the frame using a second nonlinear layer of the encoder subnetwork; Wherein the combined representation is generated based on an output of the first nonlinear layer and an output of the second nonlinear layer. The method according to claim 41, wherein the combined representation of the frame is used by the third convolutional layer of the encoder sub-network using an output of the first nonlinear layer and an output of the second nonlinear layer as input to produce. The method according to claim 32, wherein the encoded frame comprises an encoded video frame. The method according to claim 32, wherein the at least one chroma channel includes a chroma blue channel and a chroma red channel. The method according to claim 32, wherein the encoded frame has a luma-chrominance (YUV) format. A device for processing video data, comprising: a memory; and A processor, coupled to the memory and configured to: obtain an encoded frame; using a first convoluted layer of a decoder subnetwork to separate a luma channel of the encoded frame from at least one chrominance channel of the encoded frame; using a second convolutional layer of the decoder subnetwork of a neural network system to generate reconstructed output values associated with the luma channel of the encoded frame; using a third convolutional layer of the decoder subnetwork to generate reconstructed output values associated with the at least one chrominance channel of the encoded frame; and An output frame is generated, the output frame including the reconstructed output values associated with the luma channel and the reconstructed output values associated with the at least one chroma channel. The apparatus according to claim 46, wherein the first convolutional layer of the decoder sub-network comprises a 1x1 convolutional layer, the 1x1 convolutional layer comprises one or more 1x1 convolutional filters. The device according to claim 46, wherein the processor is configured to: using a first non-linear layer of the decoder subnetwork to process values associated with the luma channel of the encoded frame, wherein the reconstructed output values associated with the luma channel are based on produced by an output of the first nonlinear layer; and Values associated with the at least one chroma channel of the encoded frame are processed using a second non-linear layer of the decoder subnetwork, wherein the weighted values associated with the at least one chroma channel The structured output value is generated based on an output of the second nonlinear layer. The device according to claim 46, wherein the processor is configured to: The samples of the encoded frame are dequantized. The device according to claim 46, wherein the processor is configured to: The samples of the encoded frame are entropy decoded. The device according to claim 46, wherein the processor is configured to: Store the output frame in memory. The device according to claim 46, wherein the processor is configured to: Display the output frame. The device according to claim 46, wherein the processor is configured to: generating an output value associated with a luma channel of the frame from a first convolutional layer of an encoder sub-network of the neural network system; generating output values associated with at least one chrominance channel of the frame from a second convolutional layer of the encoder subnetwork; generating the frame by a third convolutional layer of the encoder subnetwork based on output values associated with the luma channel of the frame and output values associated with the at least one chrominance channel of the frame a combined representation; and The encoded frame is generated based on the combined representation of the frame. The apparatus according to claim 53, wherein the third convolutional layer of the encoder sub-network includes a 1x1 convolutional layer, the 1x1 convolutional layer includes one or more 1x1 convolutional filters. The device according to claim 53, wherein the processor is configured to: using a first nonlinear layer of the encoder subnetwork to process output values associated with the luma channel of the frame; and processing output values associated with the at least one chrominance channel of the frame using a second nonlinear layer of the encoder subnetwork; Wherein the combined representation is generated based on the output of the first nonlinear layer and an output of the second nonlinear layer. The apparatus according to claim 55, wherein the combined representation of the frame is used by the third convolutional layer of the encoder subnetwork as input using an output of the first nonlinear layer and an output of the second nonlinear layer to produce. The device according to claim 46, wherein the encoded frame comprises an encoded video frame. The device according to claim 57, wherein the at least one chroma channel includes a chroma blue channel and a chroma red channel. The device according to claim 46, wherein the encoded frame has a luma-chroma (YUV) format. The device according to claim 46, further comprising: at least one of a display and a camera configured to capture one or more video frames.

Images