TW202404371A - Neural network based filtering process for multiple color components in video coding - Google Patents

Abstract

A method of processing video data includes receiving a syntax element that defines a filtering mode for a neural network (NN) model for both a first color component and a second color component, applying an instance of the NN model, in the defined filtering mode, to a first block of the first color component to generate a first filtered block, and storing the first filtered block for a coding unit (CU).

Description

Neural network-based filtering procedures for multiple color components in video decoding

This patent application claims priority from U.S. Provisional Application No. 63/367,713, filed on July 5, 2022, and the entire content of this application is incorporated herein by reference.

The content of this case is about video encoding and video decoding.

Digital video capabilities can be incorporated into a variety of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablets, and e-books Readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite wireless phones, so-called "smart phones," video conference calling equipment, video streaming equipment, and more. Digital video equipment implements video decoding technologies such as MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4 Part 10, Advanced Video Coding (AVC), ITU-T Standards specified by T H.265/High Efficiency Video Coding (HEVC), ITU-T H.266/Versatile Video Coding (VVC) and extensions to these standards, as well as proprietary video transcoders/formats such as Technologies described in AOMedia Video 1 (AV1) developed by the Alliance for Open Media. Video devices can more efficiently send, receive, encode, decode and/or store digital video information by implementing these video decoding technologies.

Video decoding techniques include spatial (intra-frame) prediction and/or temporal (inter-frame) prediction to reduce or eliminate redundancy inherent in video sequences. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) can be divided into video blocks, which can also be called coding tree units (CTUs), coding units (CUs), and/or or decoding node. Video blocks in an intra-frame decoded (I) slice of a picture are encoded using spatial prediction relative to reference samples in adjacent blocks in the same picture. Video blocks in an inter-frame decoded (P or B) slice of a picture may use spatial prediction relative to reference samples in adjacent blocks in the same picture, or temporal prediction relative to reference samples in other reference pictures. The picture may be called a frame, and the reference picture may be called a reference frame.

In general, this article describes techniques used to perform filtering procedures on distorted images. The filtering procedure may be based on machine learning techniques such as neural network techniques. Example techniques may be used in the context of advanced video transcoders, such as those based on the Versatile Video Coding (VVC) standard, extensions to VVC, next-generation video coding standards, or any other video transcoder.

As described in more detail, this document describes an example of neural network-based filtering in which the color components (eg, chroma components) of video data share the same neural network pattern. For example, the video transcoder and video decoder can select the same neural network mode for the two chroma components instead of selecting different neural network modes for the two chroma components. In this way, the complexity is reduced because the video transcoder and video decoder can perform fewer filter selections and/or filter executions than if different neural network modes were selected for the two chroma components. performance and processing time.

In one example, the present disclosure describes a method of processing video data, the method including: receiving a syntax element defining a neural network (NN) for both a first color component and a second color component. ) filtering mode of the model; in the defined filtering mode, apply an instance of the NN model to the first block of the first color component to produce the first filtered block; and store the first filtering for the coding unit (CU) block.

In one example, this case content describes a device for processing video data, the device includes: a memory configured to store the video data; and one or more processors implemented using circuitry coupled to the memory , which is configured to: receive a syntax element defining a filtering mode for a neural network (NN) model for both the first color component and the second color component; in the defined filtering mode, the An instance of the NN model is applied to the first block of the first color component to produce a first filtered block; and the first filtered block is stored for a coding unit (CU).

In one example, the present disclosure describes a computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: receive syntax elements defined for A filtering mode for a neural network (NN) model for both a first color component and a second color component; in the defined filtering mode, an instance of the NN model is applied to the first block of the first color component to produce a first filtered block; and storing the first filtered block for a coding unit (CU).

The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, features and advantages will become apparent from the description, drawings and claims.

Video decoding techniques include filtering to reduce decoding artifacts. In some instances, the filtering technique may be a machine learning-based filtering technique. For example, neural networks can be utilized to perform machine learning techniques. For example, in neural network (NN) based filtering, the reconstructed samples are the input, and the intermediate outputs are the residual samples, which are added back to the input to fine-tune the input samples.

Video data can include multiple color components, such as red, green, and blue (RGB) or luminance and two chrominance components (YCbCr). In some cases, there may be different NN-based filters for each color component. However, having different NN-based filters for each color component can be computationally and processing time intensive.

In some cases, it may be possible to use a single neural network model with different NN model modes. That is, in these cases, a first filtering mode for the NN model is used to filter the first color component, and a second filtering mode for the NN model is used to filter the second color component. The video transcoder and/or video decoder may select different modes for each of the two chroma components, which may require two filterings to be performed on the decoder side. For example, in such instances, the video transcoder and video decoder may execute the NN model in a first filtering mode to filter a first block of first color components, followed by a second execution in a second filtering mode. NN model to filter the second block of the second color component. This dual execution of NN models for filtering is computationally intensive.

According to one or more examples described in this content, video transcoders and video decoders may utilize filtering control mechanisms due to similarities between different color components (e.g., similarity between two chroma components). Make two or more color components (e.g., chroma components) share the same neural network mode selection. For example, the video transcoder and video decoder may apply an instance of the NN model to the first block of the first color component in a specified filtering mode, and apply the same instance of the NN model to the first block of the first color component in a specified filtering mode. The second block of the second color component. In addition to the chroma components (e.g., the first color component and the second color component), the input to the NN model may be a luma component, but the output of the NN model may be a filtered chroma component, but not a filtered luma component (Although in some instances a filtered luminance component may be output).

To ensure that the NN model is applied to both the first block of the first color component and the second block of the second color component with the same filtering mode (e.g., where NN model filtering is enabled for both color components), The video transcoder can signal a syntax element that defines a filtering mode of the NN model for both the first color component and the second color component, and the video decoder can receive the syntax element. In this example, the video decoder may set the filtering mode for both the first color component and the second color component to the same filtering mode.

In this way, the video transcoder and video decoder can use an execution instance of the NN model to generate the filter block. Using the example techniques described in this case content, computational complexity can be reduced without compromising rate-distortion performance compared to techniques where different chroma components may use different neural network modes.

FIG. 1 is a block diagram illustrating an example video encoding and decoding system 100 that may perform the techniques described herein. The technology in this case is generally directed at decoding (encoding and/or decoding) video data. Generally, video data includes any data used to process video. Accordingly, video data may include original, unencoded video, encoded video, decoded (eg, reconstructed) video, and video relay data (such as signaling data).

As shown in FIG. 1 , in this example, system 100 includes source device 102 that provides encoded video material to be decoded and displayed by destination device 116 . Specifically, the source device 102 provides video data to the destination device 116 via the computer-readable medium 110 . Source device 102 and destination device 116 may include any of a wide range of devices, including desktop computers, notebook (e.g., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets (e.g., smart phones). telephones), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, and more. In some cases, source device 102 and destination device 116 may be equipped for wireless communications and thus may be referred to as wireless communications devices.

In the example of FIG. 1, source device 102 includes video source 104, memory 106, video transcoder 200, and output interface 108. Destination device 116 includes input interface 122, video decoder 300, memory 120, and display device 118. According to the content of this case, the video transcoder 200 of the source device 102 and the video decoder 300 of the destination device 116 may be configured to apply the following technology: machine learning-based (eg, neural network) for multiple color components in video decoding path) filtering program. Thus, source device 102 represents an instance of a video encoding device, and destination device 116 represents an instance of a video decoding device. In other examples, the source device and destination device may include other components or arrangements. For example, source device 102 may receive video material from an external video source, such as an external camera. Likewise, destination device 116 may interface with an external display device rather than include an integrated display device.

System 100 as shown in Figure 1 is only one example. In general, any digital video encoding and/or decoding device can perform the following techniques: machine learning (eg, neural network) based filtering procedures for multiple color components in video decoding. Source device 102 and destination device 116 are merely examples of such decoding devices, with source device 102 generating decoded video material for transmission to destination device 116 . This case refers to "decoding" equipment as equipment that performs decoding (encoding and/or decoding) of data. Accordingly, video transcoder 200 and video decoder 300 represent examples of decoding devices (specifically, a video transcoder and a video decoder), respectively. In some examples, source device 102 and destination device 116 may operate in a substantially symmetrical manner such that source device 102 and destination device 116 each include components for video encoding and decoding. Thus, system 100 may support one-way or two-way video transmission between source device 102 and destination device 116, for example, for video streaming, video replay, video broadcasting, or video telephony.

Typically, video source 104 represents a source of video data (i.e., raw, unencoded video data) and provides a contiguous sequence of pictures (also referred to as "frames") of the video data to video transcoder 200. The video transcoder 200 encodes image-specific data. Video source 104 of source device 102 may include a video capture device (such as a video camera), a video file containing previously captured raw video, and/or a video feed interface for receiving video from a video content provider. As a further alternative, video source 104 may generate computer graphics-based material as the source video, or a combination of live video, archived video, and computer-generated video. In each case, video transcoder 200 encodes captured, pre-captured, or computer-generated video data. Video transcoder 200 may rearrange pictures from the order in which they are received (sometimes referred to as "display order") to the decoding order for decoding. Video transcoder 200 may generate a bit stream including encoded video data. Source device 102 may then output the encoded video data onto computer-readable media 110 via output interface 108 for reception and/or retrieval via, for example, input interface 122 of destination device 116 .

Memory 106 of source device 102 and memory 120 of destination device 116 represent general purpose memory. In some examples, memory 106 , 120 may store raw video data, such as raw video from video source 104 and raw decoded video data from video decoder 300 . Additionally or alternatively, memories 106, 120 may store software instructions executable by, for example, video transcoder 200 and video decoder 300, respectively. Although memory 106 and memory 120 are shown separately from video transcoder 200 and video decoder 300 in this example, it should be understood that video transcoder 200 and video decoder 300 may also include internal memory. To achieve functionally similar or equivalent purposes. Additionally, memory 106, 120 may store encoded video data (eg, output from video transcoder 200 and input to video decoder 300). In some examples, a portion of memory 106, 120 may be allocated as one or more video buffers, for example, for storing raw, decoded and/or encoded video data.

Computer-readable media 110 may represent any type of media or device capable of transmitting encoded video material from source device 102 to destination device 116 . In one example, computer-readable media 110 represents a communication medium that enables source device 102 to transmit encoded video data directly to destination device 116 in real time, such as via a radio frequency network or a computer-based network. The output interface 108 can modulate the transmission signal including the encoded video data, and the input interface 122 can demodulate the received transmission signal according to a communication standard such as a wireless communication protocol. Communication media may include any wireless or wired communication media, such as the radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network such as a local area network, a wide area network, or a global network (eg, the Internet). The communication media may include routers, switches, base stations, or any other device that facilitates communication from source device 102 to destination device 116 .

In some examples, source device 102 may output encoded data from output interface 108 to storage device 112 . Similarly, destination device 116 may access encoded data from storage device 112 via input interface 122 . The storage device 112 may include a variety of distributed or local-access data storage media (such as hard drives, Blu-ray discs, DVDs, CD-ROMs, flash memories, volatile or non-volatile memories) or may be used to store files. Any other suitable digital storage medium for encoding video data.

In some examples, source device 102 may output the encoded video data to file server 114 or another intermediate storage device that may store the encoded video data generated by source device 102 . Destination device 116 may access stored video data from file server 114 via streaming or downloading.

File server 114 may be any type of server device capable of storing encoded video data and sending the encoded video data to destination device 116 . File server 114 may represent a network server (e.g., for a website), a server configured to provide file transfer protocol services such as File Transfer Protocol (FTP) or File Transfer One-Way (FLUTE) protocol, Content delivery network (CDN) equipment, Hypertext Transfer Protocol (HTTP) server, Multimedia Broadcast Multicast Service (MBMS) or enhanced MBMS (eMBMS) server and/or Network Attached Storage (NAS) equipment. File server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as HTTP Bearer Dynamic Self-Adapting Streaming (DASH), HTTP Real-Time Streaming (HLS), Real-time Data Streaming Protocol (RTSP), HTTP dynamic streaming and more.

Destination device 116 may access encoded video data from file server 114 via any standard data connection, including an Internet connection. This may include wireless channels (e.g., Wi-Fi connections), wired connections (e.g., Digital Subscriber Line (DSL)), cable modems, etc. suitable for accessing the encoded video data stored on file server 114 ) or a combination of both. Input interface 122 may be configured to operate in accordance with any one or more of the various protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data.

Output interface 108 and input interface 122 may represent a wireless transmitter/receiver, a modem, a wired network component (eg, an Ethernet network card), a wireless communications component operating in accordance with any of the various IEEE 802.11 standards, or Other physical parts. In instances where the output interface 108 and the input interface 122 include wireless components, the output interface 108 and the input interface 122 may be configured in accordance with cellular communications standards such as 4G, 4G-LTE (Long Term Evolution), LTE-Advanced, 5G, etc., to transmit data such as encoded video data. In some instances in which output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 may be configured to transmit signals in accordance with other wireless standards, such as the IEEE 802.11 specification, the IEEE 802.15 specification (eg, ZigBee™), the Bluetooth™ standard, etc. , to transmit data such as encoded video data. In some examples, source device 102 and/or destination device 116 may include respective system-on-chip (SoC) devices. For example, source device 102 may include an SoC device to perform functions attributed to video transcoder 200 and/or output interface 108 , and destination device 116 may include an SoC device to perform functions attributed to video decoder 300 and/or input interface 122 function.

The technology described in this case can be applied to support video decoding in any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, and Internet streaming video transmissions (e.g., HTTP-based Dynamic self-adjusting streaming (DASH), digital video encoded on data storage media), decoding of digital video stored on data storage media, or other applications.

The input interface 122 of the destination device 116 receives the encoded video bit stream from the computer-readable medium 110 (eg, communication media, storage device 112, file server 114, etc.). The encoded video bitstream may include signaling information defined by video transcoder 200 and used by video decoder 300 , such as information describing video blocks or other coding units (e.g., slices, pictures, pictures, groups, sequences, etc.) properties and syntax elements for handling values. The display device 118 displays decoded pictures of the decoded video data to the user. Display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

Although not shown in FIG. 1 , in some examples, the video transcoder 200 and the video decoder 300 may be integrated with the audio encoder and/or the audio decoder, respectively, and may include appropriate MUX-DEMUX units. or other hardware and/or software to handle multiplexed streams including both audio and video in a common data stream.

Video transcoder 200 and video decoder 300 may each be implemented as any of a variety of suitable encoder and/or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application special products. ASIC, field programmable gate array (FPGA), discrete logic, software, hardware, firmware, or any combination thereof. When these technologies are implemented partially in software, the device may store instructions for the software in an appropriate non-transitory computer-readable medium and execute these instructions in hardware using one or more processors to perform the project. Content technology. Each of the video transcoder 200 and the video decoder 300 may be included in one or more encoders or decoders, and any of the encoders or decoders may be integrated as a combined encoder in the corresponding device. / part of the CODEC. Devices including video transcoder 200 and/or video decoder 300 may include integrated circuits, microprocessors, and/or wireless communication devices (such as cellular phones).

Video transcoder 200 and video decoder 300 may be configured in accordance with ITU-T H.265 (also known as High Efficiency Video Coding (HEVC)) or extensions thereof (such as multi-view and/or scalable video coding extensions). Video decoding standards operate. Alternatively, video transcoder 200 and video decoder 300 may operate according to other proprietary or industry standards such as ITU-T H.266 (also known as Universal Video Coding (VVC)). In other examples, video transcoder 200 and video decoder 300 may be implemented in accordance with a proprietary video transcoder/format such as AOMedia Video 1 (AV1), extensions of AV1, and/or subsequent versions of AV1 (eg, AV2) operate. In other examples, video transcoder 200 and video decoder 300 may operate according to other proprietary formats or industry standards. However, the technology in this case is not limited to any specific decoding standard or format. In general, video transcoder 200 and video decoder 300 may be configured to perform the techniques described herein in conjunction with any video decoding technique that uses filtering (eg, machine learning (eg, neural network)-based filtering of video data).

Generally, video transcoder 200 and video decoder 300 may perform block-based coding of pictures. The term "chunk" generally refers to a structure (eg, encoded, decoded or other form used in an encoding and/or decoding process) containing data to be processed. For example, a block may include a two-dimensional matrix of samples of luma and/or chroma data. Generally, the video transcoder 200 and the video decoder 300 can decode video data represented in YUV (eg, Y, Cb, Cr) format. That is, instead of decoding the sampled red, green, and blue (RGB) data of the picture, the video transcoder 200 and the video decoder 300 may decode the luma and chroma components, where the chroma component may Includes both red and blue chroma components. In some examples, the video transcoder 200 converts the received RGB format data into YUV representation before encoding, and the video decoder 300 converts the YUV representation into the RGB format. Alternatively, pre- and post-processing units (not shown) can perform these transformations.

The content of this case may generally involve the decoding (eg, encoding and decoding) of pictures, to include procedures for encoding or decoding the material of the pictures. Similarly, the context may relate to the coding of blocks of pictures to include procedures for encoding or decoding data in those blocks (eg, prediction and/or residual coding). An encoded video bitstream typically includes a series of values for syntax elements that represent coding decisions (eg, coding modes) and partitioning of pictures into blocks. Therefore, a reference to a picture or block being coded should generally be understood to be the coded value of the syntax element used to form that picture or block.

HEVC defines various blocks, including coding units (CU), prediction units (PU), and transform units (TU). According to HEVC, a video decoder (such as video transcoder 200) divides coding tree units (CTUs) into CUs according to a quad-tree structure. That is, the video decoder divides the CTU and CU into four equal, non-overlapping squares, and each node of the quadtree has zero or four child nodes. A node without child nodes may be called a "leaf node", and the CU of such a leaf node may include one or more PUs and/or one or more TUs. Video decoders can further split PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents the partitioning of TUs. In HEVC, PU represents inter-frame prediction data, and TU represents residual data. An intra-predicted CU includes intra-prediction information (such as an intra-mode indication).

As another example, the video transcoder 200 and the video decoder 300 may be configured to operate according to VVC. According to VVC, a video decoder (such as video transcoder 200) divides a picture into a plurality of decoding tree units (CTUs). The video transcoder 200 may divide the CTU according to a tree structure such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure eliminates the concept of multiple partition types, such as the separation between CU, PU and TU of HEVC. The QTBT structure includes two levels: the first level divided according to quadtree partitioning, and the second level divided according to binary tree partitioning. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to coding units (CUs).

In the MTT partitioning structure, blocks can be partitioned using quadtree (QT) partitioning, binary tree (BT) partitioning, and one or more types of ternary tree (TT) (also known as ternary tree (TT)) partitioning. A ternary tree or ternary tree-type partition is a partition that splits a block into three sub-blocks. In some instances, a ternary tree or ternary tree-type partition divides the block into three sub-blocks without dividing the original block through the center. Partition types in MTT (e.g., QT, BT, and TT) can be symmetric or asymmetric.

When operating in accordance with the AV1 transcoder, video transcoder 200 and video decoder 300 may be configured to decode video material in blocks. In AV1, the largest decoding block that can be processed is called a superblock. In AV1, a superblock can be 128x128 luma samples or 64x64 luma samples. However, in subsequent video coding formats (eg, AV2), super-blocks may be defined via different (eg, larger) luma sample sizes. In some instances, the superblock is the top level of a block quadtree. Video transcoder 200 may further partition the super-block into smaller decoding blocks. Video transcoder 200 may partition super-blocks and other coding blocks into smaller blocks using square or non-square partitioning. Non-square blocks may include N/2xN, NxN/2, N/4xN, and NxN/4 blocks. Video transcoder 200 and video decoder 300 may perform separate prediction and transformation procedures for each coding block.

AV1 also defines tiles for video data. A tile is a rectangular array of superblocks that can be decoded independently of other tiles. That is, the video transcoder 200 and the video decoder 300 can respectively encode and decode the decoding blocks within a tile without using video data from other tiles. However, video transcoder 200 and video decoder 300 may perform filtering across tile boundaries. Tiles can be uniform or non-uniform in size. Tile-based coding enables parallel processing and/or multiple threads for encoder and decoder implementations.

In some examples, video transcoder 200 and video decoder 300 may use a single QTBT or MTT structure to represent each of the luma and chrominance components, while in other examples, video transcoder 200 and video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT/MTT structure for the luma component and another QTBT/MTT structure for the two chrominance components (or two QTBT/MTT structures for the corresponding chroma component).

The video transcoder 200 and the video decoder 300 may be configured to use quad-tree partitioning, QTBT partitioning, MTT partitioning, super-block partitioning, or other partitioning structures.

In some examples, the CTU includes a coding tree block (CTB) of luma samples, two corresponding CTBs of chroma samples for a picture with three sample arrays, or a monochrome picture or using three independent color planes and using The CTB of the sample in the picture that is used to decode the syntax structure for decoding the sample. A CTB can be an NxN sample block of some N value such that the partitioning of components into CTBs is a partitioning. A component is an array or a single sample from one of the three arrays (luma and two chroma) that make up a picture in a 4:2:0, 4:2:2, or 4:4:4 color format, or a single sample that makes up a picture. An array within an array or a single sample of a color format picture. In some examples, the coding block is an MxN sample block for some M and N values of samples, such that the partitioning of CTBs into coding blocks is a partitioning.

Blocks (eg, CTUs or CUs) can be grouped in a picture in various ways. As an example, a brick can represent a rectangular area with CTU rows within a specific tile in the image. A tile can be a rectangular area with CTU within a specific tile column and specific tile row in the picture. A tile column refers to a CTU of a rectangular area with a height equal to the height of the picture and a width specified by a syntax element (eg, such as in a picture parameter set). A tile row refers to a CTU of a rectangular area with a height specified by a syntax element (eg, such as in a picture parameter set) and a width equal to the width of the picture.

In some examples, a tile may be divided into multiple bricks, each of which may include one or more CTU rows within the tile. Tiles that are not divided into multiple bricks can also be called bricks. However, bricks, which are a true subset of tiles, cannot be called tiles. You can also arrange the bricks in the picture into slices. A slice can be an integer number of bricks in a picture that can be specifically contained within a single Network Abstraction Layer (NAL) unit. In some instances, a slice includes multiple complete tiles, or a contiguous sequence of complete bricks including just one tile.

This content can use "NxN" and "N by N" interchangeably to represent the sampling size of a block (such as a CU or other video block) in terms of vertical and horizontal dimensions, such as 16x16 samples or 16x16 samples. Typically, a 16x16 CU will have 16 samples in the vertical direction (y=16), and 16 samples in the horizontal direction (x=16). Likewise, a NxN CU typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU can be arranged by rows and columns. Furthermore, a CU does not have to have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU may contain NxM samples, where M is not necessarily equal to N.

Video transcoder 200 encodes video data for CUs representing prediction and/or residual information and other information. Prediction information indicates how a CU is to be predicted in order to form a prediction block for that CU. Residual information usually represents the sample-by-sample difference between the samples of the CU before encoding and the prediction block.

To predict a CU, video transcoder 200 may typically form prediction blocks for the CU via inter prediction or intra prediction. Inter-frame prediction usually means predicting a CU from data of a previously decoded picture, while intra-frame prediction usually means predicting a CU from previously decoded data of the same picture. To perform inter-frame prediction, video transcoder 200 may use one or more motion vectors to generate prediction blocks. Video transcoder 200 may typically perform a motion search to identify a reference block that closely matches the CU, such as from the perspective of differences between the CU and the reference block. Video transcoder 200 may calculate the difference metric using sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean square error (MSD), or other such difference calculations to determine Whether the reference block closely matches the current CU. In some examples, video transcoder 200 may use unidirectional prediction or bidirectional prediction to predict the current CU.

Some examples of VVC also provide an affine motion compensation mode, which can be considered an inter-frame prediction mode. In affine motion compensation mode, video transcoder 200 may determine two or more motion vectors that represent non-translational motion, such as zoom-in or zoom-in, rotation, perspective motion, or other irregular motion types.

To perform intra prediction, video transcoder 200 may select an intra prediction mode to generate prediction blocks. Some instances of VVC offer 67 intra-frame prediction modes, including various directional modes as well as planar and DC modes. Typically, video transcoder 200 selects an intra prediction mode, which describes predicting samples of the current block from adjacent samples relative to adjacent samples of the current block (eg, a block of a CU). Assuming that video transcoder 200 decodes CTUs and CUs in raster scan order (left to right, top to bottom), these samples can typically be above, above, and left of the current block in the same picture as the current block. Or the left side.

Video transcoder 200 encodes data indicating the prediction mode used for the current block. For example, for inter prediction modes, video transcoder 200 may encode data indicating which of various available inter prediction modes to use, as well as motion information for the corresponding mode. For unidirectional or bidirectional inter-frame prediction, for example, the video transcoder 200 may use advanced motion vector prediction (AMVP) or merge mode to encode motion vectors. Video transcoder 200 may use a similar mode to encode motion vectors for affine motion compensation mode.

AV1 includes two common technologies for encoding and decoding coding blocks of video material. The two common techniques are intra-frame prediction (eg, intra-frame prediction or spatial prediction) and inter-frame prediction (eg, inter-frame prediction or temporal prediction). In the context of AV1, when using intra-frame prediction mode to predict blocks of the current frame of video data, video transcoder 200 and video decoder 300 do not use video data from other frames of video data. For most intra-frame prediction modes, video transcoder 200 encodes blocks of the current frame based on differences between sample values in the current block and prediction values generated from reference samples in the same frame. The video transcoder 200 determines prediction values based on the reference samples based on the intra prediction mode.

After prediction, such as intra prediction or inter prediction for a block, video transcoder 200 may calculate residual data for the block. Residual information, such as a residual block, represents the sample-by-sample difference between a block and a predicted block formed for that block using the corresponding prediction mode. Video transcoder 200 may apply one or more transforms to the residual block to produce transformed data in the transform domain rather than the sample domain. For example, video transcoder 200 may apply a discrete cosine transform (DCT), integer transform, wavelet transform, or conceptually similar transform to the residual video data. Additionally, the video transcoder 200 may apply a secondary transform after the first transform, such as mode-dependent non-separable secondary transform (MDNSST), signal-dependent transform, Karhunen-Loeve transform (KLT), etc. The video transcoder 200 generates transform coefficients after applying the one or more transforms.

As mentioned above, after performing any transformation to generate transform coefficients, the video transcoder 200 may perform quantization of the transform coefficients. Quantization generally represents the procedure of quantizing transform coefficients to possibly reduce the amount of data used to represent them, thus providing further compression. By performing a quantization procedure, video transcoder 200 may reduce the bit depth associated with some or all of these transform coefficients. For example, the video transcoder 200 may round an n- bit value to an m -bit value during quantization, where n is greater than m . In some examples, to perform quantization, video transcoder 200 may perform a bitwise right shift of the value to be quantized.

After quantization, the video transcoder 200 may scan the transform coefficients and generate a one-dimensional vector based on the two-dimensional matrix including the quantized transform coefficients. The sweep can be designed to place higher energy (and therefore lower frequency) transform coefficients at the front of the vector, and lower energy (and therefore higher frequency) transform coefficients at the back of the vector. In some examples, video transcoder 200 may scan the quantized transform coefficients using a predefined scan order to generate a serialized vector, and then entropy encode the quantized transform coefficients of the vector. In other examples, video transcoder 200 may perform self-adjusting scanning. After scanning the quantized transform coefficients to form a one-dimensional vector, the video transcoder 200 may entropy encode the one-dimensional vector, such as context-based self-adaptive binary arithmetic coding (CABAC). The video transcoder 200 may also entropy encode the value of a syntax element that describes relay data associated with the encoded video data for use by the video decoder 300 when decoding the video data.

To perform CABAC, video transcoder 200 may assign context within the context model to the symbols to be sent. For example, the context may relate to whether a symbol's adjacent value is zero. Probabilistic decisions can be based on the context assigned to the symbol.

The video transcoder 200 may also generate information specific to the video decoder in, for example, picture headers, block headers, segment headers or other syntax data (such as sequence parameter set (SPS), picture parameter set (PPS) or video parameter set (VPS)). 300 grammar materials (such as block-based grammar materials, picture-based grammar materials, and sequence-based grammar materials). The video decoder 300 can similarly decode such syntax data to determine how to decode the corresponding video data.

In this manner, video transcoder 200 may generate a bitstream that includes encoded video data, such as syntax elements describing partitioning of a picture into blocks (eg, CUs), and prediction and/or prediction of the blocks. or syntax element for residual information. Finally, the video decoder 300 can receive the bit stream and decode the encoded video data.

Typically, the video decoder 300 executes a process identical to that executed by the video transcoder 200 to decode the encoded video data of the bit stream. For example, video decoder 300 may use CABAC to decode the values of syntax elements of the bit stream in a manner that is substantially similar to (albeit mutually) similar to the CABAC encoding process of video transcoder 200 . The syntax element may define partitioning information for partitioning the picture into CTUs and partitioning each CTU according to a corresponding partitioning structure (such as a QTBT structure) to define CUs of the CTU. Syntax elements may further specify prediction and residual information for blocks (eg, CUs) of video data.

The residual information may be represented via, for example, quantized transform coefficients. Video decoder 300 may inverse-quantize and inverse-transform the quantized transform coefficients of a block to reproduce a residual block of the block. Video decoder 300 uses the signaled prediction mode (intra-frame or inter-frame prediction) and the associated prediction information (eg, motion information for inter-frame prediction) to form a prediction block for the block. Video decoder 300 may then combine the prediction block and the residual block (on a sample-by-sample basis) to reproduce the original block. Video decoder 300 may perform other processing, such as deblocking to reduce visual artifacts along block boundaries.

The content of this case usually involves "signaling" certain information (such as grammatical elements). The term "signaling" may generally refer to transmitting values for syntax elements and/or other data used to decode encoded video data. That is, video transcoder 200 may signal the values for the syntax elements in the bitstream. Typically, signaling means producing a value in a bit stream. As mentioned above, source device 102 may stream bits to destination device 116 substantially instantaneously or non-immediately, such as when storing syntax elements to storage 112 for later retrieval by destination device 116 . This happens.

As mentioned above, video coding standards include ITU-T H.261, ISO/IEC MPEG-1 visual, ITU-T H.262 or ISO/IEC MPEG-2 visual, ITU-TH.263, ISO/ IEC MPEG-4 Vision and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), High Efficiency Video Coding (HEVC), or ITU-T H.265, including its range extensions, multi-view extensions ( MV-HEVC) and scalable extensions (SHVC). Recently, the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG) developed the latest standard Universal Video Coding (VVC) or ITU-T H. 26. Version 1 of the VVC specification, hereafter referred to as VVC FDIS, was finalized this month and is available at http://phenix.int-evry.fr/jvet/doc_end_user/documents/19_Teleconference/wg11/JVET-S2001-v17. zip obtained.

The video coding standard is based on the so-called hybrid video coding principle, as shown in Figure 6 and similar to the description of Figure 2. The term hybrid represents the combination of two techniques to reduce redundancy in video signals, such as quantized prediction with prediction residuals and transform coding. Prediction and transform reduce redundancy in the video signal through decorrelation, while quantization reduces the data represented by the transform coefficients by reducing their accuracy, ideally by removing only irrelevant details. This hybrid video decoding design principle is also used in the two latest standards, HEVC and VVC. As shown in Figure 6, and also described in more detail with reference to Figure 2, modern hybrid video decoders consist of various processes.

As shown in Figure 6, modern hybrid video decoders 130 typically perform block partitioning, motion compensation or inter-picture prediction, intra-picture prediction, transform, quantization, entropy coding, and post-/in-loop filtering. In the example of FIG. 2 , the video decoder 130 includes a summation unit 134 , a transformation unit 136 , a quantization unit 138 , an entropy decoding unit 140 , an inverse quantization unit 142 , an inverse transform unit 144 , a summation unit 146 , and a loop filtering unit 148 , decoded picture buffer (DPB) 150 , intra prediction unit 152 , inter prediction unit 154 and motion estimation unit 156 .

Generally, the video decoder 130 may receive input video data 132 when encoding the video data. Block partitioning is used to divide the picture (image) of the received video data into smaller blocks for the operation of prediction and transformation procedures. Early video coding standards used a fixed block size, usually 16×16 samples. Recent standards, such as HEVC and VVC, employ tree-based partitioning structures to provide flexible partitioning.

Motion estimation unit 156 and inter-frame prediction unit 154 may predict input video data 132 based on previously decoded data of DPB 150, for example. Motion compensation or inter-picture prediction exploits the redundancy that exists between pictures in a video sequence (hence "inter-picture"). According to block-based motion compensation used in all modern video transcoders, predictions are obtained from one or more previously decoded pictures (ie, reference pictures). The corresponding region for generating inter-frame prediction is indicated via motion information including a motion vector and a reference picture index. In recent video transcoders, a hierarchical prediction structure within a group of pictures (GOP) is applied to improve decoding efficiency. An example of a GOP 700 having a size equal to 16 is illustrated in FIG. 7 .

Summing unit 134 may calculate residual data as the difference between input video data 132 and prediction data from intra prediction unit 152 or inter prediction unit 154 . Summing unit 134 provides the residual block to transform unit 136, which applies one or more transforms to the residual block to produce a transform block. Quantization unit 138 quantizes the transform blocks to form quantized transform coefficients. Entropy coding unit 140 entropy encodes the quantized transform coefficients and other syntax elements such as motion information or intra-prediction information to produce an output bit stream 158 .

At the same time, the inverse quantization unit 142 inversely quantizes the quantized transform coefficients, and the inverse transform unit 144 inversely transforms the transform coefficients to reproduce the residual block. Summing unit 146 combines the residual block with the prediction block (on a sample-by-sample basis) to produce a decoded block of the video material. Loop filtering unit 148 applies one or more filters (eg, neural network-based filters, neural network-based loop filters, neural network-based back-loop filters, self-adjusting loop filters, or at least one of a predefined self-adjusting loop filter) to produce a filtered decoding block.

In accordance with the techniques of this disclosure, the neural network filtering unit of loop filtering unit 148 may be derived from summing unit 146 and from one or more other units of hybrid video decoder 130 (e.g., transform unit 136, quantization unit 138, in-frame One or more other filtering units within prediction unit 152, inter-frame prediction unit 154, motion estimation unit 156, and/or loop filtering unit 148) receive data for decoded pictures of video data. For example, the neural network filtering unit may receive data from the deblocking filtering unit (also referred to as the “deblocking unit”) of the loop filtering unit 148 .

In Figure 6, intra-picture prediction exploits the spatial redundancy in the picture memory (hence "intra-frame prediction") by deriving predictions of blocks from spatially adjacent (reference) samples that have been encoded/decoded. In the latest video transcoders (including AVC, HEVC and VVC), directional angle prediction, DC prediction, and planar or planar prediction are used.

Transform: The hybrid video coding standard applies block transformations to prediction residuals (whether they come from inter-picture prediction or intra-picture prediction). In early standards, including H.261/262/263, the discrete cosine transform (DCT) was used. In HEVC and VVC, in addition to DCT, more transformation cores are also applied to consider different statistical characteristics in specific video signals.

Quantization is intended to reduce the precision of an input value or set of input values in order to reduce the amount of data required to represent those values. In hybrid video coding, quantization is typically applied to the residual samples of each transform (ie, the transform coefficients), resulting in integer coefficient levels. In recent video coding standards, the step size is derived according to a so-called quantization parameter (QP) that controls fidelity and bit rate. Larger step sizes reduce bit rate, but also reduce quality, which can cause video images to exhibit blocking artifacts and blurry details, for example.

Entropy Coding: Contextual Self-Adjusting Binary Arithmetic Coding (CABAC) is used in recent video transcoders (e.g., AVC, HEVC, and VVC) due to its high efficiency.

Post-/in-loop filtering is a filtering procedure (or a combination of these procedures) applied to reconstruct a picture to reduce decoding artifacts. The input to the filtering process is usually a reconstructed image, which is a combination of the reconstructed residual signal (including quantization error) and the prediction. That is, the input to the filtering procedure is the sum of the prediction and residual signals. As shown in Figure 6, the reconstructed picture after in-loop filtering is stored and used as a reference for inter-picture prediction of subsequent pictures. Decoding artifacts are mainly determined through QP, so QP information is usually used for the design of filter procedures. In HEVC, in-loop filtering includes deblocking filtering and sample self-adjusting offset (SAO) filtering. In the VVC standard, the self-adjusting loop filter (ALF) is introduced as the third filter. The filtering program of ALF is as follows, (1)

where container is the sample before the filtering procedure, is the sample value after the filtering procedure. represents the filter coefficient, is the clipping function, and Indicates clipping parameters. The variables k and l are in and varies between, where L represents the filter length. clipping function , which corresponds to the function . The clipping operation introduces nonlinearity, making ALF more efficient by reducing the impact of adjacent sample values that are too different from the current sample value. In VVC, filter parameters can be signaled in the bit stream and can be selected from a predefined set of filters. The ALF filtering procedure can also be summarized as the following equation. (2)

Neural network (NN) based filtering for video decoding is described below. Embedding neural networks into the hybrid video decoding framework can improve compression efficiency. Neural networks have been used in modules such as intra-frame prediction and inter-frame prediction to improve prediction efficiency. NN based in-loop filters are also possible. In some instances, the filtering procedure is applied as a post-filter, in which case the filtering procedure is applied only to the output picture and the unfiltered picture is used as the reference picture.

In addition to existing filters such as deblocking filters, SAO and ALF, NN-based filters can also be additionally applied. It can also be used in specialized applications where it is designed to replace all existing filters.

As shown in Figure 8, the NN-based filtering procedure takes reconstructed samples 800 as input, and the intermediate outputs are residual samples (eg, residual samples 802) that are added back to the input (eg, reconstructed samples 800) to fine-tune the input samples and produce filtered samples 804. NN filters can use all color components as input to exploit cross-component correlation. Different components can share the same filter (including network structure and model parameters), or each component can have its own specific filter.

As an example, a NN filter can receive all color components even if a subset of the color components is being filtered. For example, for luma and chroma components, a NN filter can receive both luma and chroma blocks. In this example, it is assumed that the Cb chrominance block is filtered and the luma or Cr chrominance blocks are not filtered. The luma block, Cb chroma block and Cr chroma block are all inputs to the NN filter. In such instances, the NN filter may produce filtered Cb chroma blocks and filtered Cr chroma blocks. Filtered luma blocks may not be generated, but these techniques require that filtered luma blocks be generated. However, video transcoder 200 and video decoder 300 may discard the filtered Cr chroma blocks. That is, a reconstructed coding unit (CU) including a luma block and a chroma block may include reconstructed luma samples (unfiltered), reconstructed Cr chroma samples (unfiltered), and filtered Reconstructed Cb chroma sampling.

The filtering procedure can also be summarized as follows: (3)

The model structure and model parameters of the NN-based filter may be predefined and stored at the video transcoder 200 and the video decoder 300 . These filters can also be signaled in the bit stream.

When applying NN-based filtering in video decoding, the entire video signal can be divided into multiple processing components, and each processing component can be processed independently. Possible choices for processing components include: frames, slices/tiles, CTUs, or any predefined or signaled shape and size. That is, the term "processing component" can refer to the portion of an image or frame where processing may occur.

NN-based filtering with multi-modal design is described below. In order to further improve the performance of NN-based filtering, multi-mode solutions can be designed. For example, for each processing component, video transcoder 200 may select among a set of modes based on rate-distortion optimization, and may signal the selection in the bitstream. Different modes may include different NN models, different values used as input information for the NN model, etc. To give an example, JVET-Z0113: Y. Li, K. Zhang, L. Zhang, H. Wang, M. Coban, A. M. Kotra, M. Karczewicz, F. Galpin, K. Andersson, J. StröM, D. Liu , R.Sjöberg, EE1-1.7: Combined Testing of EE1-1.6 and EE1-1.3, JVET-Z0113, April 2022 (hereinafter referred to as JVET-Z0113), proposes a NN-based filtering solution via Multiple modes are built based on a single NN model using different QP values as input to NN models for different modes.

There may be certain problems using NN-based filtering techniques. For video signals with multiple color components (eg, video data), the filtering procedures for different color components may be different. However, for some types of filtering (e.g., neural network-based filtering), applying a different filter to each color component can significantly increase computational complexity and memory requirements. In these cases, a better trade-off between compression performance and complexity cost can be achieved by designing filters that cover multiple color components and having appropriate control over how these filters are applied in the video transcoder. .

For example, in JVET-Z0113, a NN-based filtering solution with multiple modes is proposed, which uses different QP values for different modes. The video transcoder 200 and the video decoder 300 can target processing components. Choose from different modes. In this technique of JVET-Z0113, only one model (e.g., only one neural network model) can be used for filtering of the chroma component of a slice (e.g., where a slice is an instance of a processing component). The two chroma components can choose different modes, which makes it necessary to perform two filterings on the decoder side for the inference of these two chroma components.

In some technologies (such as those of JVET-Z0113), if the color components have different modes of the NN filter, the video transcoder 200 and the video decoder 300 will perform the NN filter in the corresponding different modes. Two examples. For a different mode, in JVET-Z0113 the inputs are luma and chroma samples, but the output can be just chroma samples for a NN filter designed to filter the chroma components. That is, the NN filtering mode for the chroma component can use the luma component as input, but not the output luma samples. For example, assume that a first NN filtering mode is applied to a first color component (eg, Cb), and a second NN filtering mode is applied to a second color component (eg, Cr).

In this example, video transcoder 200 and video decoder 300 will apply a first instance of the NN filter in a first NN filtering mode, with inputs being luma, Cb, and Cr components (eg, luma block, Cb block, and Reconstructed samples of Cr blocks). Video transcoder 200 and video decoder 300 will ignore the filtered Cr samples and maintain the filtered Cb samples.

Next or in parallel, video transcoder 200 and video decoder 300 will apply a second instance of the NN filter in a second NN filtering mode with the inputs being luma, Cb, and Cr components (eg, luma block, Cb block and reconstruction sampling of Cr blocks). Video transcoder 200 and video decoder 300 will ignore the filtered Cb samples and maintain the filtered Cr samples.

In this example, video transcoder 200 and video decoder 300 may maintain the filtered Cb samples from the first instance of the NN filter, maintain the filtered Cr samples from the second instance of the NN filter, and discard the remaining samples. . Therefore, in this example, to produce filtered CUs that include luma and chroma components, video transcoder 200 and video decoder 300 apply two instances of NN filters in different modes. This execution of two instances of NN filters in different modes can be processing intensive and delay image reconstruction.

A CU is an instance of a composite block that includes component blocks. For example, a CU includes luma blocks and chroma blocks. In this context, CU is used as an example, which generally refers to a composite block that includes different color components (such as luma and chroma components). The term CU shall not be limited to CUs as defined in video coding standards, or as part of a video transcoder, unless explicitly stated otherwise.

This content describes example techniques that restrict different modes for different color components. That is, the NN filtering mode for one color component is the same as the filtering mode for the other color component. In this way, the video transcoder 200 and the video decoder 300 can apply one instance of the NN filter in different modes instead of applying two instances of the NN filter.

Additionally, this document describes example techniques that allow for selectively deciding whether to utilize filtered sampling from multiple color components or fewer color components (eg, one color component). In this way, all filtered components do not need to be used, allowing the situation to result in better decoding efficiency or higher image content by utilizing a subset of filtered components.

Additionally, as described in greater detail, this document describes example techniques for signaling different levels of parameters to indicate which NN filtering mode will be used. For example, a slice-level syntax element can have a plurality of values. The first value may indicate that NN filtering is disabled for the color component. The value subsets of the plurality of values may each correspond to a different NN filtering mode. The last value may indicate whether NN filtering is enabled or disabled, and if enabled, indicates the NN filtering mode at a lower level (eg, at the CTU level for all blocks of the CTU, at the CU level, etc.). In such instances, video transcoder 200 may signal, and video decoder 300 may receive, additional syntax elements that indicate whether NN filtering is enabled or disabled, and if If enabled, indicates NN filtering mode.

In one or more of the examples described in this content, due to the similarity between the two chroma components, a filtering control mechanism can be designed so that the chroma components share the same NN mode selection to achieve the desired performance without sacrificing rate-distortion performance. Reduce computational complexity as much as possible. That is, this disclosure describes example techniques for filtering procedures in which some or all, if any, of the filtering operations are shared by multiple color components.

The following describes a first implementation of the example technology described in the content of this case. As an implementation, for color formats with one luma component and two chroma components (e.g., YCbCr color format), at most one NN-based filter is allowed for each processing component to complete the filtering of two chroma components. program.

In a first example, a NN filter for chroma color components may be designed to be able to output two filtered chroma components after a single filtering procedure. Filter control of the two chrominance components can be performed jointly. For example, both chroma components have exactly the same choices for how to apply the NN filter, including the on/off switch for NN filtering, the input data for how to build the NN model, and the values of all input elements so that they can be used in a single After the filtering procedure, filtered samples of the two chroma components are obtained.

Similar to JVET-Z0113, for the chroma component, there is a model for NN filtering, and in the procedure of establishing the input of the NN model, multiple modes are established by using different QP values. The choice of NN filter configuration is signaled and controlled as follows:

At the slice level (as one example of a processing component, but other examples are possible), a syntax element with a value range of [0,4] is signaled for the chroma component (called the "first-level filter mode syntax element" ”). The following lists the meanings of the 5 syntax values: a. 0: For this slice, turn off the NN filtering of the chroma component. b. 1: For the chroma component of this slice, turn on NN filtering, and in the procedure that creates the input to the NN filter, use the first QP in the predefined or signaled set. c. 2: For the chroma component of this slice, turn on NN filtering, and in the procedure that creates the input to the NN filter, use the second QP from the predefined or signaled set. d. 3: For the chroma component of this slice, turn on NN filtering, and in the procedure that creates the input to the NN filter, use the third QP from the predefined or signaled set. e. 4: Send additional syntax elements to control NN filtering individually for each processing component.

If mode 4 is signaled at the slice level, then for each processing component, an additional syntax element with a value in the range [0,3] for the corresponding chroma component is signaled (e.g., a "second-level filtering mode syntax element") . The following lists the meanings of the 4 syntax values: a. 0: For this processing component, turn off the NN filtering of the chroma component. b. 1: For the chrominance component of this processing component, NN filtering is turned on, and in the procedure that creates the input to the NN filter, the first QP in the predefined or signaled set is used. c. 2: For the chrominance component of the processed component, turn on NN filtering, and in the procedure that creates the input to the NN filter, use the second QP from the predefined or signaled set. d. 3: For the chroma component of the processed component, turn on NN filtering, and in the procedure that creates the input to the NN filter, use the third QP from the predefined or signaled set.

For example, in the above example, video transcoder 200 may signal a syntax element (eg, a first-level filter mode syntax element) that defines the usage of both the first color component and the second color component. The filtering mode is based on a neural network (NN) model, and the video decoder 300 can receive the syntax element. For example, in the above example, the value of the first-level syntax element is in the range [0,4]. If the value of the first-level syntax element is 1, 2, or 3, the first-level syntax element defines the filtering mode of the NN model (eg, NN filtering mode) for both the first color component and the second color component. For example, a value of 1 means that the filtering mode is based on the first QP in a predefined or signaled set, a value of 2 means that the filtering mode is based on the second QP in a predefined or signaled set, and a value of 3 means that the filtering mode is based on the first QP in a predefined or signaled set. The pattern is based on a third QP in a predefined or signaled set.

However, if the value of the first-level filter mode syntax element is 4, the video transcoder 200 may signal a syntax element (eg, a second-level filter mode syntax element) that defines the syntax for the first color component. and the second color component, and the video decoder 300 may receive the syntax element. For example, in the above example, the value of the second-level syntax element is in the range [0,3]. If the value of the first-level syntax element is 1, 2, or 3, the second-level syntax element defines the filtering mode (eg, NN filtering mode) of the NN model for both the first color component and the second color component. For example, a value of 1 means that the filtering mode is based on the first QP in a predefined or signaled set, a value of 2 means that the filtering mode is based on the second QP in a predefined or signaled set, and a value of 3 means that the filtering mode is based on the second QP in a predefined or signaled set. Based on a third QP in a predefined or signaled set. That is, video transcoder 200 may signal a syntax element that defines a filtering pattern for the NN model for both the first color component and the second color component, and video decoder 300 may receive the syntax element. In one example, the syntax element is a first level filter mode syntax element. In another example, the syntax element is a second level filter mode syntax element. For example, video transcoder 200 may signal a first syntax element (eg, a first-level filter mode syntax element) that is applicable to a plurality of CUs (eg, a sliced CU), and video decoder 300 may receive the A first syntax element, wherein the first syntax element indicates that the second syntax element is parsed for a subset of CUs in the plurality of CUs (e.g., the first level filtering mode syntax element has a value of 4, so the second level filtering mode syntax elements are parsed). The CU subset includes one or more CUs with the CU being reconstructed. That is, first-level filtering mode syntax elements apply to more CUs than second-level filtering mode syntax elements.

Video transcoder 200 may parse the second syntax element for the CU subset based on the first syntax element indication to signal the second syntax element, and video decoder 300 may receive the second syntax element. That is, under the condition that the value of the first-level filtering mode syntax element is 4, the video transcoder 200 can signal the second-level filtering mode syntax element for the CU subset, and the video decoder 300 can receive .

Examples with first-level filtering mode syntax elements and second-level filtering mode syntax elements are provided for illustrative purposes only and should not be considered limiting. That is, in some examples, there may be one filter mode syntax element that defines the filter mode of the CU. This filter mode syntax element can be at the picture level (e.g., for all CUs of the picture), at the slice level (e.g., for all CUs of the slice), at the CTU level (e.g., for all CUs of the CTU), or At the CU level (e.g., block-by-block).

Video transcoder 200 and video decoder 300 may apply an instance of the NN model to the first block of the first color component under a defined filtering mode to produce a first filtered block. For example, the first block may be part of the reconstructed samples 800 of FIG. 8 and the first residual value may be the value of the residual samples 802 of FIG. 8 . Video transcoder 200 and video decoder 300 may generate a first filtered block (e.g., using at the filtered samples of the first color component 804).

In one or more instances, since a syntax element (eg, a first-level filtering mode syntax element or a second-level filtering mode syntax element) defines a NN model for both the first color component and the second color component, filtering mode, so video transcoder 200 and video decoder 300 may apply the same instance of the NN model to the second block of the second color component in the defined filtering mode to produce a second filtered block residual value. For example, video transcoder 200 and video decoder 300 may apply the same instance of the NN model to a second block of the second color component in a defined filtering mode to produce a second residual value. Video transcoder 200 and video decoder 300 may generate a second filtered block based on the second block and the second residual value (eg, adding the second block and the second residual value).

In this context, unless otherwise stated, applying the same instance of a NN model means that one execution of the NN model results in filtered sampling for multiple color components. For example, a NN filter can receive luma samples, Cb samples, and Cr samples as input. The NN filtering mode may be defined via syntax elements (eg, a value of 1, 2, or 3 for the first level filtering mode syntax element or for the second level filtering mode syntax element). Video transcoder 200 and video decoder 300 may apply instances of NN filters in defined filtering modes, and the outputs may be filtered Cb samples and filtered Cr samples.

In order to describe that the filtered Cb samples and the filtered Cr samples are generated according to the same instance of applying the NN model, the content of this case describes the video transcoder 200 and the video decoder 300 as: (1) Under the defined filtering mode, apply an instance of the NN model to the first block of the first color component to produce the first filtered block (e.g., based on the first block and the first residual value), and (2) in the defined filtering mode, The same instance of the NN model is applied to the second block of the second color component to produce a second filtered block (eg, based on the second block and the second residual value). That is, only one instance of the NN model is executed to produce a first filtered block for the first color component and a second filtered block for the second color component.

Thus, in this example, one syntax element (eg, a first-level filtering mode syntax element or a second-level filtering mode syntax element) defines a filtering mode for two chroma components. In some other technologies (eg, Z-0133), one syntax element may define the filtering mode for the first color component, and another syntax element may define the filtering mode for the second color component. Using the example technique described in this case content, a syntax element defines a filtering mode for a NN model for multiple color components.

In the above example, video transcoder 200 and video decoder 300 may store sample values for a decoding unit (CU) based on the first filtered block and the second filtered block. For example, the video transcoder 200 and the video decoder 300 may store the first filtered block and the second filtered block as part of the CU in their respective buffers, such that if the CU is used for inter-frame prediction, the first filtered block is used. The first filtered block and the second filtered block are used for inter-frame prediction. As another example, the video decoder 300 may store the first filtered block and the second filtered block in the buffer as part of the CU, so that when a picture including the CU is displayed, the first filtered block and the second filtered block are displayed based on the first filtered block and the second filtered block. Filter blocks are used to generate image content.

As mentioned before, one syntax element defines the filtering mode of the NN model for both the first color component and the second color component (eg, NN filtering mode for NN filter). In some examples, both the first filtered block for the first color component and the second filtered block for the second color component are stored for the CU. However, in some examples, both the first filtered block and the second filtered block may not be stored for the CU.

It should be noted that in some examples, the first filtered block for the first color component or the second filtered block for the second color component may be generated even if one of the first filtered block for the first color component or the second filtered block for the second color component is not stored for the CU. filtered block and a second filtered block. For example, assume that the filtered Cb component is required, but the filtered Cr component is not required for this CU. In this example, the NN model can receive both luma and chroma components and produce a filtered Cb component and a filtered Cr component. However, the video transcoder 200 and the video decoder 300 may discard the filtered Cr component. To store values for CUs, video transcoder 200 and video decoder 300 may store the original values of the filtered Cb and Cr components (eg, unfiltered reconstructed samples).

Example techniques for selecting which filtered components are used to store values for a CU are described below. For ease of description, this case content describes the filter control used to switch the NN filter on or off. Such revelation of whether the NN filter is on or off can be seen in the example of whether filtered color components are used. A filtered color component may still be produced, but in instances where filtering is turned off for that color component, the filtered color component is discarded.

The above example of storing a first filtered block for a first color component and a second filtered block for a second color component for a CU if filtering is enabled is referred to as the first example technique. That is, in the first example technique, one syntax element defines the filtering mode for the NN model for both the first color component and the second color component. Furthermore, in this first example technique, video transcoder 200 and video decoder 300 store a first filtered block for a first color component and a second filtered block for a second color component as CU-specific value. That is, the video transcoder 200 and the video decoder 300 may apply the same instance of the NN model to the first block of the first color component and the second block of the second color component in a defined filtering mode to generate respectively A first filtered block and a second filtered block.

For example, the video transcoder 200 and the video decoder 300 may apply the same instance of the NN model to a first block of a first color component and a second block of a second color component under a defined filtering mode to produce a second block of a second color component, respectively. a residual value and a second result value. Video transcoder 200 and video decoder 300 may generate a first filtered block based on the first block and the first residual value (eg, via adding them together), and a second filtered block based on the second block and the second residual value. values to produce the second filtered block (eg, by adding them together).

In the second example, similar to the first example, the NN filter for the chroma color components is designed to be able to output both chroma components after a single filtering procedure. The filtering control allows the on/off switching of the NN filter for the two chroma components to be signaled separately. Additionally, if the NN filtering for the two chroma components is on, the output of the two chroma components should be passed through A single filter program is generated. That is, both filtered chroma blocks may be the output of the NN filter, but at least one filtered chroma block may be discarded.

At the slice level (as an example of a processing component), a syntax element with a value range of [0,4] (called first_chroma_mode_slice in the following description) is signaled for the first chroma component. For the second chroma component, if first_chroma_mode_slice==0, then another syntax element with a value range of [0,4] for the second chroma component (referred to as second_chroma_mode_slice in the description below) is signaled, to indicate the selection of NN filtering for the second chroma component. In other cases (first_chroma_mode_slice !=0), a flag with a value in the range [0,1] is signaled, and if the flag is 0, second_chroma_mode_slice is set to 0, otherwise second_chroma_mode_slice is set to first_chroma_mode_slice.

The meanings of the five syntax values of first_chroma_mode_slice and second_chroma_mode_slice are the same as in the first example, except that the range of each mode can be limited to the corresponding chroma component. That is, the values of first_chroma_mode_slice and second_chroma_mode_slice may be the same as those described above for the first-level filtering mode syntax element or the second-level filtering mode syntax element. The logic of decoding/exporting first_chroma_mode_slice and second_chroma_mode_slice can be summarized as the following pseudocode. In the pseudocode below, the syntax elements encoded in the bitstream are provided in bold and all syntax values are assumed to be 0 before the decoding/export logic begins. first_chroma_mode_slice if (first_chroma_mode_slice == 0) { second_chroma_mode_slice } else { second_chroma_flag_slice if (second_chroma_flag_slice == 1) { second_chroma_mode_slice = first_chroma_mode_slice } else { second_chroma_mode_slice = 0 } }

Similar to the first example where there is a first-level filtering mode syntax element and a second-level filtering mode syntax element, additional syntax is signaled if first_chroma_mode_slice or second_chroma_mode_slice is equal to 4 (e.g., the first-level filtering mode syntax element has a value of 4) Element (e.g., second-level filter mode syntax element) for selecting NN filtering separately for each processing component. The decoding/exporting logic can be summarized in the pseudocode below. Syntax elements decoded in the bitstream are provided in bold, and all syntax values are assumed to be 0 before the decoding/export logic begins. if (first_chroma_mode_slice == 4 || second_chroma_mode_slice == 4) { if(first_chroma_mode_slice == 0) first_chroma_mode_processing_unit = 0 else first_chroma_mode_processing_unit if (second_chroma_mode_slice == 0) second_chroma_mode_processing_unit = 0 else if (first_chroma_mode_processing_unit != 0) { second_chroma_flag_processing_unit if (second_chroma_flag_processing_unit == 1) second_chroma_mode_processing_unit = first_chroma_mode_processing_unit else second_chroma_mode_processing_unit = 0 } else second_chroma_mode_processing_unit }

In this example, the interpretation of the slice level and processing into hierarchical mode values is the same as the explanation above for the first level filtering mode syntax element and the second level filtering mode syntax element. At the slice level, 0 means NN filtering is off, 1 to 3 correspond to the 3 QP choices that can be used in the procedure to build the input information for the NN model, and 4 means at the processing component level (e.g., in all QPs used for CTB). CTB level of the CU, at the CU level, etc.) signals additional information to indicate the NN filter selection for each processing component. Similarly, in the processing hierarchy, 0 indicates that NN filtering is turned off, and 1 to 3 correspond to the 3 QP choices that can be used in the process of establishing input information for the NN model.

Thus, in one or more examples, syntax elements signaled by video transcoder 200 and received by video decoder 300 may have a value indicating that NN-based filtering is enabled for the first color component (eg, first_chroma_mode_slice is true). In this example, video transcoder 200 may signal a flag indicating that NN-based filtering is enabled for the second color component (eg, second_chroma_flag_slice is true), and video decoder 300 may receive the flag (eg, second_chroma_flag_slice). In this example, video transcoder 200 and video decoder 300 may generate a second filter based on the second block of the second color component by applying the same instance of the NN model to the second block in a defined filtering mode. filter block. To store the CU-specific value, video transcoder 200 and video decoder 300 may store the CU-specific value based on the first filtered block and the second filtered block.

In some examples, the filtered second color component may not be used as the value for the CU. For example, video transcoder 200 may signal a flag indicating that NN-based filtering is disabled for the second color component (eg, second_chroma_flag_slice is false), and video decoder 300 may receive the flag (eg, second_chroma_flag_slice). In this example, video transcoder 200 and video decoder 300 may be based on a first filtered block of a first color component (e.g., using a NN filtered block) and a second block of a second color component (e.g., not using NN filtered block) to store the value for CU.

In some examples, the first color component may not be filtered (eg, first_chroma_mode_slice==0). In this example, video transcoder 200 may signal another syntax element (eg, second_chroma_mode_slice) indicating the filtering mode for the second color component, and video decoder 300 may receive it. In this example, video transcoder 200 and video decoder 300 may be based on a first block for a first color component (e.g., a block not filtered using NN) and a second filtered block for a second color component (e.g., , blocks filtered using NN) to store values for CU.

Generally, for the second example, the first_chroma_mode_slice syntax element can be considered to be a syntax element that defines a filtering mode for a neural network (NN) model for both the first color component and the second color component. For example, if the first_chroma_mode_slice syntax element is non-zero, NN filtering is enabled for the first color component, and second_chroma_flag_slice indicates that filtering is enabled for the second color component, then first_chroma_mode_slice defines for both the first and second color components Which filtering mode. Even in instances where second_chroma_flag_slice is false, first_chroma_mode_slice can be considered to define the filtering mode for both the first and second color components, since no additional syntax element indicating the filtering mode is required.

As a third example, the third example is the same as the second example, except that an alternative way of decoding/exporting NN filter selections is used. For each color component, flags indicating on/off selection are decoded, and if at least one of the flags indicates that NN filtering is on, additional syntax elements are signaled. Slice-level (as an example of a processing component, but other examples are possible) decoding is summarized below. Syntax elements decoded in the bitstream are provided in bold, and all syntax values are assumed to be 0 before the decoding/export logic begins. first_chroma_flag_slice second_chroma_flag_slice if (first_chroma_flag_slice || second_chroma_flag_slice) { chroma_mode_slice if(first_chroma_flag_slice==1) first_chroma_mode_slice = chroma_mode_slice + 1 if(second_chroma_flag_slice==1) second_chroma_mode_slice = chroma_mode_slice + 1 }

In this instance, first_chroma_flag_slice and second_chroma_filg_slice have a range value of [0,1], and chroma_mode_slice has a range of [0,3], so the value ranges of first_chrome_mode_stice and second_chrome_mode_slice remain the same as in the second instance. The range [0,4] and numerical interpretation are also consistent. That is, chroma_mode_slice can be considered an example syntax element that defines a filtering mode for a neural network (NN) model for both the first color component and the second color component. The value used for chroma_mode_slice can be the same as the first-level filter mode syntax element described above.

For example, video transcoder 200 may signal a flag to enable NN-based filtering for the first color component, and video decoder 300 may receive it. In this example, video transcoder 200 may signal first_chroma_flag_slice that enables NN-based filtering for the first color component, and video decoder 300 may receive it. Similarly, video transcoder 200 may signal a flag to enable NN-based filtering for the second color component, and video decoder 300 may receive it. In this example, video transcoder 200 may signal second_chroma_flag_slice that enables NN-based filtering for the second color component, and video decoder 300 may receive it.

Video transcoder 200 may signal a syntax element that defines a NN model for both the first color component and the second color component based on a flag indicating that NN-based filtering is enabled for the first color component. In the filtering mode, the video decoder 300 can receive the syntax element. For example, video transcoder 200 may signal a chroma_mode_slice syntax element based on first_chroma_flag_slice being true, and video decoder 300 may receive the syntax element. Similarly, video transcoder 200 may signal a syntax element that defines usage for both the first color component and the second color component based on a flag indicating that NN-based filtering is enabled for the second color component. The filtering mode of the NN model is used, and the video decoder 300 can receive the syntax element. For example, video transcoder 200 may signal a chroma_mode_slice syntax element based on second_chroma_flag_slice being true, and video decoder 300 may receive the syntax element.

If NN filtering is enabled (eg, turned on) for one or both of the first and second color components, the video decoder 300 may determine the filtering mode based on the value of chroma_mode_slice. As shown above, the equation that determines the filter mode used for both color components is the same (for example, the filter mode used for both color components is chroma_mode_slice+1). Therefore, chroma_mode_slice defines the filtering mode for the NN model for both the first color component and the second color component.

The equivalent of this instance can be established by modifying the way in which grammatical elements are derived and interpreted. In the following examples, examples of this equivalent design are described.

In this example, if first_chroma_mode_slice or second_chroma_mode_slic equals 4, then additional syntax elements are signaled for the NN filtering selection respectively for each processing component (eg, similar to the second-level filter mode syntax element). The decoding/exporting logic can be summarized as pseudocode. Syntax elements encoded in the bitstream are provided in bold, and all syntax values are assumed to be 0 before the decoding/export logic begins. if (first_chroma_mode_slice == 4 || second_chroma_mode_slice == 4) { if (first_chroma_mode_slice == 0) first_chroma_mode_processing_unit = 0 else first_chroma_flag_processing_unit if (second_chroma_mode_slice == 0) second_chroma_mode_processing_unit = 0 else second_chroma_flag_processing_unit if (first_chroma_flag_processing_unit != 0 || first_chroma_flag_processing_unit != 0) chroma_mode_processing_unit if (first_chroma_flag_processing_unit == 1) first_chroma_mode_processing_unit = chroma_mode_processing_unit+1 else first_chroma_mode_processing_unit = 0 if (second_chroma_flag_processing_unit == 1) second_chroma_mode_processing_unit = chroma_mode_processing_unit + 1 else second_chroma_mode_processing_unit = 0 }

In this example, first_chroma_flag_processing_unit and second_chroma_flag_processing_unit have a range value of [0,1], and chroma_mode_processing_unit has a range value of [0,2], so the value ranges of first_chroma_mode_processing_unit and second_chroma_flag_processing_unit remain the same as in the second example. The range [0,3] and numerical interpretation are also consistent. The equivalent of this instance can be established by modifying the way in which syntax elements are derived and interpreted. Examples of such equivalent designs are described in later examples.

As a fourth instance, the fourth instance is equivalent to the third instance, the derivation/interpretation of the syntax values is different, but the actual NN control is the same. Slice-level decoding can be summarized as follows. Syntax elements decoded in the bitstream are provided in boldface, and all syntax values are assumed to be initialized to 0 before the decoding/export logic begins. first_chroma_flag_slice second_chroma_flag_slice if (first_chroma_flag_slice || second_chroma_flag_slice) { chroma_mode_slice if(first_chroma_flag_slice==1) first_chroma_mode_slice = chroma_mode_slice if(second_chroma_flag_slice==1) second_chroma_mode_slice = chroma_mode_slice }

In this case, the range values of first_chroma_flag_slice and second_chroma_flag_slice are [0,1], and the range value of chroma_mode_slice is [0,3]. As a result, the value range of first_chroma_mode_slice and second_chroma_mode_slice becomes [0,3]. This instance is equivalent to the third instance according to the following explanation: a. If first_chroma_flag_slice==0, turn off NN filtering for the first chroma component. b. If first_chroma_flag_slice==1, further signal/export the NN filter selection, as follows: i. If first_chroma_mode_slice==0, then in the procedure to create the input of the NN filter, use the first QP in the predefined or signaled set. ii. If first_chroma_mode_slice==1, then in the procedure that creates the input of the NN filter, use the second QP from the predefined or signaled set. iii. If first_chroma_mode_slice==2, then in the procedure to create the input of the NN filter, use the third QP from the predefined or signaled set. iv. If first_chroma_mode_slice==4, send additional syntax elements to control NN filtering for each processing component separately.

Similar to the slice level, when NN filtering selection is signaled at the processing level, first_chroma_mode_processing_unit and second_chroma_mode_processing_unit are assigned a chroma_mode_processing_unit when NN filtering is turned on (as shown in the pseudocode below). Syntax elements decoded in the bitstream are provided in boldface, and all syntax values are assumed to be initialized to 0 before the decoding/export logic begins. if (first_chroma_mode_slice == 4 || second_chroma_mode_slice == 4) { if (first_chroma_mode_slice == 0) first_chroma_mode_processing_unit = 0 else first_chroma_flag_processing_unit if (second_chroma_mode_slice == 0) second_chroma_mode_processing_unit = 0 else second_chroma_flag_processing_unit if (first_chroma_flag_processing_unit != 0 || first_chroma_flag_processing_unit != 0) chroma_mode_processing_unit if (first_chroma_flag_processing_unit) first_chroma_mode_processing_unit = chroma_mode_processing_unit else first_chroma_mode_processing_unit = 0 if (second_chroma_flag_processing_unit) second_chroma_mode_processing_unit = chroma_mode_processing_unit else second_chroma_mode_processing_unit = 0 }

The interpretation can be the same as at the slice level, except that the mode value 4 may not be present in the processing component level: a. If first_chroma_flag_processing_unit==0, turn off NN filtering for the first chroma component. b. If first_chroma_flag_processing_unit==1, then further signal/export the NN filter selection, as follows: i. If first_chroma_mode_processing_unit==0, then in the procedure of establishing the input of the NN filter, the first QP in the predefined or signaled set is used. ii. If first_chroma_mode_processing_unit==1, then in the procedure of establishing the input of the NN filter, use the second QP from the predefined or signaled set. iii. If first_chroma_mode_processing_unit==2, then in the procedure of establishing the input of the NN filter, use the third QP from the predefined or signaled set.

The first to fourth examples can be extended to any multi-mode NN filtering design, and the example techniques described in the content of this case can still be applied. For example, in one or more instances, a syntax element defining a filtering mode for a NN model for both a first color component and a second color component, defining which parameter to select from a plurality of parameters, and defined Parameters define the filtering mode. That is, a syntax element defining a filtering mode may mean that the syntax element defines which parameter is to be used, and that parameter defines the filtering mode.

For example, the syntax element may have a value of 1, 2, or 3, where a value of 1 indicates that the first parameter from a predefined set or a signaled set defines the filtering mode. A value of 2 indicates that a second parameter from a predefined set or a signaled set defines the filtering mode, and a value of 3 indicates that a third parameter from a predefined set or a signaled set defines the filtering mode. In some instances, the parameters are quantized parameters.

Give some examples of additional parameters or collections from which parameters are selected: a. Instead select the QP value from a predefined/signaled set with size 3. The method can be extended to the case where a candidate set of N values is used. b. A multi-modal NN filtering method can be designed using any value used in the program that builds the input of the NN filtering model, instead of using different QP values for different modes. c. Multi-mode design can use different NN models for different modes instead of using different QP values for different modes. d. The different ways of designing multi-mode filtering methods described above can be combined to create more complex multi-mode designs. For example, these techniques can be combined to build M models, and for each model there are N QP values that can be used when building the inputs to the model. Subsequently, a total of M*N patterns were established. A 4-model design can be established by using two models and selecting two QPs for each model.

As a second implementation, the example technique of the first implementation can be extended to cover more color components. For example, all instances of the first implementation can be designed so that the NN model can output all three color components (RGB, YUV, etc.), and the filtering controls described in this case can be used to perform at most once for all three color components. NN filtering operation.

FIG. 2 is a block diagram illustrating an example video transcoder 200 that may perform the techniques of this disclosure. Figure 2 is provided for purposes of explanation and should not be considered a limitation on the techniques broadly illustrated and described in this context. For ease of explanation, the content of this case describes the video transcoder 200 based on VVC technology (ITU-T H.266 under development) and HEVC (ITU-T H.265) technology. However, the techniques of this disclosure may be performed by video encoding devices configured to implement other video encoding standards and video encoding formats, such as AV1 and its successors to the AV1 video encoding format.

In the example of FIG. 2, the video transcoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transformation processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transformation processing unit 212, Reconstruction unit 214, filtering unit 216, decoded picture buffer (DPB) 218, and entropy encoding unit 220. Video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, DPB 218 and entropy Any or all of the encoding units 220 may be implemented in one or more processors or in processing circuitry. For example, units of the video transcoder 200 may be implemented as one or more circuits or logic components, as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. Additionally, video transcoder 200 may include supplemental or alternative processors or processing circuitry to perform these and other functions.

Video data memory 230 may store video data to be encoded by components of video transcoder 200 . Video transcoder 200 may receive video data stored in video data memory 230 from, for example, video source 104 (FIG. 1). The DPB 218 may serve as a reference picture memory that stores reference video data for use by the video transcoder 200 when predicting subsequent video data. Video data memory 230 and DPB 218 may be composed of a variety of storage devices such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other type of storage device). Video data memory 230 and DPB 218 may be provided by the same storage device or different storage devices. In various examples, video data memory 230 may be on-chip with other components of video transcoder 200, as shown, or off-chip relative to those components.

In the context of this case, references to video data memory 230 should not be construed as being limited to memory internal to video transcoder 200 (unless so specifically described), nor should it be construed as being limited to memory outside video transcoder 200 of memory (unless so specifically described). Rather, a reference to the video data memory 230 should be understood as a reference memory that stores the video data that the video transcoder 200 receives for encoding (eg, the video data for the current block to be encoded). Memory 106 of FIG. 1 may also provide temporary storage of outputs from various units of video transcoder 200.

The various units of FIG. 2 are illustrated to illustrate understanding the operations performed by the video transcoder 200. These units may be implemented as fixed function circuits, programmable circuits, or a combination thereof. Fixed function circuits represent circuits that provide a specific function and are preset in terms of the operations they can perform. Programmable circuits represent circuits that can be programmed to perform a variety of tasks and provide flexible functionality in the operations that can be performed. For example, the programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits can execute software instructions (for example, for receiving parameters or outputting parameters), but the types of operations performed by fixed-function circuits are generally immutable. In some examples, one or more of the units may be distinct circuit blocks (fixed function or programmable), and in some examples, the one or more units may be integrated circuits.

The video transcoder 200 may include an arithmetic logic unit (ALU), an elementary functional unit (EFU), a digital circuit, an analog circuit, and/or a programmable core formed of programmable circuits. In examples where software executed by programmable circuitry is used to perform operations of video transcoder 200 , memory 106 ( FIG. 1 ) may store instructions (eg, object code) for the software that video transcoder 200 receives and executes. , or another memory (not shown) in the video transcoder 200 can store such instructions.

The video data memory 230 is configured to store received video data. The video transcoder 200 can retrieve the picture of the video data from the video data memory 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data memory 230 may be original video data to be encoded.

Mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra prediction unit 226. The mode selection unit 202 may include other functional units to perform video prediction according to other prediction modes. For example, mode selection unit 202 may include palette units, intra-block replication units (which may be part of motion estimation unit 222 and/or motion compensation unit 224), affine units, linear model (LM) units, and the like. .

Mode selection unit 202 typically coordinates multiple encoding passes to test combinations of encoding parameters and resulting rate-distortion values for such combinations. Coding parameters may include: CTU to CU partitioning, prediction mode for CU, transform type for residual data of CU, quantization parameters for residual data of CU, etc. The mode selection unit 202 may ultimately select a combination of encoding parameters that has a better rate-distortion value than other tested combinations.

The video transcoder 200 may divide the picture retrieved from the video data memory 230 into a series of CTUs, and encapsulate one or more CTUs in a segment. The mode selection unit 202 may divide the CTU of the picture according to a tree structure (such as the MTT structure, QTBT structure, super-block structure, or quad-tree structure described above). As mentioned above, the video transcoder 200 may form one or more CUs by dividing the CTU according to a tree structure. Such CUs may also be commonly referred to as "video blocks" or "blocks".

Typically, mode select unit 202 also controls its components (eg, motion estimation unit 222, motion compensation unit 224, and intra prediction unit 226) to generate overlapping PUs and TUs for the current block (eg, the current CU, or in HEVC, PU and TU). part) prediction block. For inter-frame prediction of the current block, motion estimation unit 222 may perform a motion search to identify one or more reference pictures (e.g., one or more previously coded pictures stored in DPB 218) or Multiple closely matched reference blocks. Specifically, the motion estimation unit 222 may calculate, for example, based on the sum of absolute differences (SAD), the sum of squared differences (SSD), the mean absolute difference (MAD), the mean square error (MSD), etc., to represent the difference between the potential reference block and the current How similar values do blocks have. Motion estimation unit 222 may typically perform these calculations using sample-by-sample differences between the current block and the reference block under consideration. Motion estimation unit 222 may identify the reference block with the minimum value produced by these calculations, which reference block indicates the reference block that most closely matches the current block.

Motion estimation unit 222 may form one or more motion vectors (MVs) that define the location of the reference block in the reference picture relative to the current block in the current picture. Motion estimation unit 222 may then provide the motion vector to motion compensation unit 224. For example, for unidirectional inter-frame prediction, motion estimation unit 222 may provide a single motion vector, and for bi-directional inter-frame prediction, motion estimation unit 222 may provide two motion vectors. Motion compensation unit 224 may then use the motion vectors to generate prediction blocks. For example, motion compensation unit 224 may use motion vectors to retrieve reference block information. As another example, if the motion vector has fractional sampling precision, motion compensation unit 224 may interpolate the value for the prediction block according to one or more interpolation filters. In addition, for bidirectional inter-frame prediction, the motion compensation unit 224 may retrieve data of two reference blocks identified by corresponding motion vectors, for example, via sample-by-sample averaging or weighted averaging, and combine the retrieved data.

When operating in accordance with the AV1 video coding format, motion estimation unit 222 and motion compensation unit 224 may be configured to use translational motion compensation, affine motion compensation, overlapping block motion compensation (OBMC), and/or composite intra-frame prediction, Encodes coding blocks of video data (eg, luma and chroma coding blocks).

As another example, for intra prediction or intra prediction coding, intra prediction unit 226 may generate a prediction block based on samples adjacent to the current block. For example, for directional mode, intra prediction unit 226 may typically mathematically combine values from adjacent samples and pad these calculated values in a defined direction across the current block to produce a prediction block. As another example, for DC mode, intra prediction unit 226 may calculate an average of adjacent samples to the current block and generate a prediction block to include the resulting average for each sample of the prediction block.

When operating according to the AV1 video coding format, the intra prediction unit 226 may be configured to use directional intra prediction, non-directional intra prediction, recursive filtered intra prediction, chroma from luma (CFL) Prediction, intra-block copy (IBC), and/or palette modes encode coding blocks of video data (eg, luma and chroma coding blocks). The mode selection unit 202 may include additional functional units to perform video prediction according to other prediction modes.

The mode selection unit 202 supplies the prediction block to the residual generation unit 204. The residual generation unit 204 receives the original unencoded version of the current block from the video data memory 230 and the prediction block from the mode selection unit 202 . The residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block for the current block. In some examples, the residual generation unit 204 may also determine the difference between the sample values in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). In some examples, residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.

In examples where mode selection unit 202 partitions a CU into PUs, each PU may be associated with a luma prediction unit and a corresponding chroma prediction unit. The video transcoder 200 and the video decoder 300 can support PUs of various sizes. As mentioned above, the size of the CU may represent the size of the luma coding block of the CU, and the size of the PU may represent the size of the luma prediction unit of the PU. Assuming that the size of a specific CU is 2Nx2N, the video transcoder 200 can support a PU size of 2Nx2N or NxN for intra-frame prediction, and support a symmetric PU size of 2Nx2N, 2NxN, Nx2N, NxN, etc. for intra-frame prediction. prediction. The video transcoder 200 and the video decoder 300 may also support asymmetric partitioning of PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter-frame prediction.

In instances where mode selection unit 202 does not further partition the CU into PUs, each CU may be associated with a luma coding block and a corresponding chroma coding block. As mentioned above, the size of the CU may represent the size of the luma decoding block of the CU. The video transcoder 200 and the video decoder 300 may support a CU size of 2Nx2N, 2NxN or Nx2N.

For other video coding techniques, such as intra-block copy mode coding, affine mode coding, and linear model (LM) mode coding, to name some examples, mode selection unit 202 selects Each unit generates a prediction block for the current block being encoded. In some examples (such as palette mode coding), mode selection unit 202 may not generate predictive blocks but instead generate syntax elements that indicate how the blocks are reconstructed based on the selected palette. In such modes, mode selection unit 202 may provide these syntax elements to entropy encoding unit 220 for encoding.

As mentioned above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. Subsequently, the residual generation unit 204 generates a residual block for the current block. To generate a residual block, residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

Transform processing unit 206 applies one or more transforms to the residual block to produce a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transforms to the residual block to form a block of transform coefficients. For example, transform processing unit 206 may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform to the residual block. In some examples, transform processing unit 206 may perform multiple transforms (eg, a primary transform and a secondary transform (eg, a rotation transform)) on the residual block. In some examples, transform processing unit 206 does not apply transforms to the residual blocks.

When operating in accordance with AV1, transform processing unit 206 may apply one or more transforms to the residual block to produce a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transforms to the residual block to form a block of transform coefficients. For example, transform processing unit 206 may apply a horizontal/vertical transform combination, which may include a discrete cosine transform (DCT), an asymmetric discrete sine transform (ADST), a flipped ADST (eg, inverse ADST), and an identity transform (IDTX) . When using an identity transformation, the transformation is skipped in one of the vertical or horizontal directions. In some instances, transformation processing may be skipped.

Quantization unit 208 may quantize the transform coefficients in the block of transform coefficients to produce a quantized block of transform coefficients. Quantization unit 208 may quantize the transform coefficients of the block of transform coefficients based on a quantization parameter (QP) value associated with the current block. Video transcoder 200 (eg, via mode selection unit 202) may adjust the degree of quantization applied to the block of transform coefficients associated with the current block by adjusting the QP value associated with the CU. Quantization may result in loss of information, and therefore, the accuracy of the quantized transform coefficients may be lower than the accuracy of the original transform coefficients generated by the transform processing unit 206 .

The inverse quantization unit 210 and the inverse transform processing unit 212 may respectively apply inverse quantization and inverse transform to the quantized transform coefficient block to reconstruct the residual block from the transform coefficient block. Reconstruction unit 214 may generate a reconstructed block corresponding to the current block (although possibly with a certain degree of distortion) based on the reconstructed residual block and the prediction block generated by mode selection unit 202 . For example, reconstruction unit 214 may add samples of the reconstructed residual block to corresponding samples in the prediction block generated by mode selection unit 202 to produce a reconstructed block.

Filtering unit 216 may perform one or more filtering operations on the reconstructed blocks. For example, filtering unit 216 may perform deblocking operations to reduce blocking artifacts along edges of CUs. In some examples, operation of filtering unit 216 may be skipped.

According to one or more examples described in this content, the filtering unit 216 may be configured to perform neural network-based filtering techniques. For example, filtering unit 216 may perform neural network-based filtering techniques in addition to or instead of other filtering techniques such as ALF and SAO. For example, the filtering unit 216 may be part of a reconstruction loop that includes the decoding process performed by the video transcoder 200 . The filtering unit 216 may perform the neural network-based filtering described in this context as part of the in-loop filtering in the reconstruction loop of the decoding process of the video transcoder 200 .

When operating according to AV1, filtering unit 216 may perform one or more filtering operations on the reconstructed blocks. For example, filtering unit 216 may perform deblocking operations to reduce blocking artifacts along CU edges. In other examples, filtering unit 216 may apply constrained direction enhancement filtering (CDEF), which may be applied after deblocking, and may include applying non-separable, non-linear low-pass directional filtering based on the estimated edge directions. The filtering unit 216 may also include a loop recovery filter applied after the CDEF, and may include a separable symmetric normalized Wiener filter or a dual self-steering filter.

Video transcoder 200 stores the reconstructed blocks in DPB 218. For example, in instances where the operations of filtering unit 216 are not performed, reconstruction unit 214 may store the reconstructed blocks into DPB 218 . In an example in which the operations of filtering unit 216 are performed, filtering unit 216 may store the filtered reconstructed blocks into DPB 218 . Motion estimation unit 222 and motion compensation unit 224 may retrieve reference pictures from DPB 218, which are formed from reconstructed (and possibly filtered) blocks for inter-frame prediction of subsequently encoded pictures. Additionally, intra prediction unit 226 may use reconstructed blocks in DPB 218 of the current picture to perform intra prediction on other blocks in the current picture.

Generally, entropy encoding unit 220 may entropy encode syntax elements received from other functional components of video transcoder 200 . For example, entropy encoding unit 220 may entropy encode the quantized transform coefficient block from quantization unit 208. As another example, entropy encoding unit 220 may entropy encode prediction syntax elements from mode selection unit 202 (eg, motion information for inter prediction or intra mode information for intra prediction). Entropy encoding unit 220 may perform one or more entropy encoding operations on syntax elements that are another instance of video data to generate entropy-encoded data. For example, entropy encoding unit 220 may perform context self-adjusting variable length coding (CAVLC) operations, CABAC operations, variable-to-variable (V2V) length coding operations, syntax-based context self-adjusting binary arithmetic coding (SBAC) operations , a Probabilistic Interval Partitioning Entropy (PIPE) decoding operation, an exponential gorilla coding operation, or another type of entropy coding operation on the data. In some examples, entropy encoding unit 220 may operate in a bypass mode that does not entropy encode syntax elements.

Video transcoder 200 may output a bitstream that includes entropy-encoded syntax elements required for reconstructing blocks of segments or pictures. Specifically, the entropy encoding unit 220 may output a bit stream.

According to AV1, the entropy encoding unit 220 may be configured as a symbol-to-symbol self-adjusting multi-symbol arithmetic decoder. A grammar element in AV1 consists of an alphabet of N elements, while a context (e.g., a probabilistic model) consists of a set of N probabilities. Entropy encoding unit 220 may store the probabilities as n-bit (eg, 15-bit) cumulative distribution functions (CDFs). Entropy encoding unit 22 may perform recursive scaling using an update factor based on the alphabet size to update the context.

The operations described above are described with respect to blocks. Such descriptions should be understood to be for the operation of the luma coding block and/or the chroma coding block. As mentioned previously, in some examples, the luma coding block and the chroma coding block are the luma and chroma components of the CU. In some examples, the luma coding block and the chroma coding block are the luma and chroma components of the PU.

In some examples, operations performed for the luma coding block need not be repeated for the chroma coding block. As an example, the operation of identifying motion vectors (MVs) and reference pictures for luma coding blocks does not need to be repeated to identify MVs and reference pictures for chroma blocks. Instead, the MV for the luma coding block can be scaled to determine the MV for the chroma block, and the reference pictures can be the same. As another example, the intra prediction process may be the same for luma coding blocks and chroma coding blocks.

Video transcoder 200 represents an example of a device configured to encode video data. The device includes a memory configured to store video data, and one or more processing units implemented using circuitry. The one or more processing units The processing unit is configured to: filter two or more color components of the video material using a NN-based filter from a single neural network (NN) model, and to use the NN-based filter in a single filter program The two or more color components are then output. In some examples, filtering includes: deciding how to apply a NN-based filter, whether to turn a NN-based filter on or off, how to create input data for a single NN model, and targeting two or more color components and Determine the values of all input elements in the same way.

In some examples, video transcoder 200 may be configured to signal information indicating whether a NN-based filter is on or off for each of two or more color components. In such instances, filtering includes filtering based on whether a NN-based filter is on or off for each of the two or more color components.

For example, video transcoder 200 may signal syntax elements that define filtering modes for a neural network (NN) model for both the first color component and the second color component. Examples of syntax elements include the first-level filtering mode syntax element, the second-level filtering mode syntax element as described above, first_chroma_mode_slice (e.g., where first_chroma_mode_slice is non-zero, and second_chroma_flag_slice is true), and chroma_mode_slice (e.g., where first_chroma_flag_slice and/or second_chroma_flag_slice is true).

The filtering unit 216 may be configured to apply an instance of the NN model to the first block of the first color component in a defined filtering mode to produce a first filtered block. For example, filtering unit 216 may generate first residual samples using reconstructed samples of a first color component (eg, a first chroma block) and add the first residual samples to the reconstructed samples of the first color component. Samples are taken to produce a first filtered block.

The filtering unit 216 may be configured to apply the same instance of the NN model to the second block of the second color component in a defined filtering mode to produce a second filtered block. For example, filtering unit 216 may use the reconstructed samples of the second color component (eg, the second chroma block) to generate second residual samples and add the second residual samples to the reconstructed samples of the second color component. Samples are taken to produce a second filtered block.

In one or more examples, filtering unit 216 may store the first filtered block and the second filtered block. For example, the sample values that filtering unit 216 stores in DPB 218 for a CU may be a first filtered block and a second filtered block. In this way, if the CU is used for inter-frame prediction, the filtered block will be used for inter-frame prediction.

In some cases, filtering unit 216 may store the second filtered block only when filtering is enabled for the second color component. For example, filtering unit 216 may generate a first filtered block and a second filtered block for the same (eg, single) instance of the NN model. However, sometimes the second filtered block may not be stored (eg, discarded).

As an example, video transcoder 200 may signal the second_chroma_flag_slice flag described above as true to indicate that NN-based filtering is enabled for the second color component. In this example, filtering unit 216 may store the value for the CU based on the first filtered block (for the first color component) and the second filtered block (for the second color component). In another example, video transcoder 200 may signal the second_chroma_flag_slice flag described above as false to indicate that NN-based filtering is disabled for the second color component. In this example, filtering unit 216 may store values for the CU based on the first filtered block (for the first color component) and the second block (for the second color component) without filtering. Again, the second filtered block may be generated via applying an instance of the NN model; however, when the second_chroma_flag_slice flag is false, the second filtered block may be discarded.

FIG. 3 is a block diagram illustrating an example video decoder 300 that may perform the techniques of this disclosure. Figure 3 is provided for purposes of explanation and should not be considered a limitation on the techniques broadly illustrated and described in this context. For ease of explanation, the content of this case describes the video decoder 300 based on VVC technology (ITU-T H.266 under development) and HEVC technology (ITU-T H.265). However, the techniques described in this case may be performed by video decoding devices configured to implement other video decoding standards.

In the example of FIG. 3, the video decoder 300 includes a coded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, Filter unit 312 and decoded picture buffer (DPB) 314. Any one or all of the CPB memory 320, the entropy decoding unit 302, the prediction processing unit 304, the inverse quantization unit 306, the inverse transform processing unit 308, the reconstruction unit 310, the filtering unit 312 and the DPB 314 may be one or more implemented in a processor or in processing circuitry. For example, the units of the video decoder 300 may be implemented as one or more circuits or logic components, as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. Additionally, video decoder 300 may include supplemental or alternative processors or processing circuitry to perform these and other functions.

Prediction processing unit 304 includes motion compensation unit 316 and intra prediction unit 318. Prediction processing unit 304 may include other units for performing predictions according to other prediction modes. For example, prediction processing unit 304 may include palette units, intra-block copy units (which may form part of motion compensation unit 316), affine units, linear model (LM) units, and the like. In other examples, video decoder 300 may include more, fewer, or different functional components.

When operating in accordance with AV1, compensation unit 316 may be configured to use translational motion compensation, affine motion compensation, OBMC, and/or composite inter-intra prediction to code blocks of video data (e.g., luma and color decoding block), as described above. Intra-prediction unit 318 may be configured to use directional intra-prediction, non-directional intra-prediction, recursive filtered intra-prediction, CFL, intra-block copy (IBC), and/or palette mode, Decode the coding blocks of the video data (eg, luma and chroma coding blocks) as described above.

CPB memory 320 may store video data (such as an encoded video bit stream) to be decoded by components of video decoder 300 . For example, the video data stored in the CPB memory 320 can be obtained from the computer readable medium 110 (FIG. 1). CPB memory 320 may include a CPB that stores encoded video data (eg, syntax elements) from the encoded video bit stream. Furthermore, CPB memory 320 may store video data in addition to the syntax elements of the coded picture, such as temporary data representing the output from the various units of video decoder 300 . The DPB 314 typically stores decoded pictures, and the video decoder 300 may output and/or use the decoded pictures as reference video data when decoding subsequent data or pictures in the encoded video bit stream. CPB memory 320 and DPB 314 may be formed from any of a variety of storage devices, such as DRAM (including SDRAM), MRAM, RRAM, or other types of storage devices. CPB memory 320 and DPB 314 may be provided by the same storage device or different storage devices. In various examples, CPB memory 320 may be on-die with other components of video decoder 300 or off-die relative to those components.

Additionally or alternatively, in some examples, video decoder 300 may retrieve decoded video data from memory 120 (FIG. 1). That is, memory 120 may store data, as discussed above with reference to CPB memory 320 . Likewise, memory 120 may store instructions to be executed by video decoder 300 when some or all of the functions of video decoder 300 are implemented using software executed by the processing circuitry of video decoder 300 .

The various units of FIG. 3 are illustrated to illustrate understanding the operations performed by the video decoder 300. These units may be implemented as fixed function circuits, programmable circuits, or a combination thereof. Similar to Figure 2, a fixed-function circuit represents a circuit that provides a specific function and is preset in terms of the operations it can perform. Programmable circuits represent circuits that can be programmed to perform a variety of tasks and provide flexible functionality in the operations that can be performed. For example, the programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits can execute software instructions (for example, for receiving parameters or outputting parameters), but the types of operations performed by fixed-function circuits are generally immutable. In some examples, one or more of the units may be distinct circuit blocks (fixed function or programmable), and in some examples, the one or more units may be integrated circuits.

Video decoder 300 may include ALU, EFU, digital circuits, analog circuits, and/or a programmable core formed of programmable circuits. In instances where operations of video decoder 300 are performed via software executing on programmable circuitry, on-chip or off-chip memory may store instructions (eg, object code) for the software that video decoder 300 receives and executes.

Entropy decoding unit 302 may receive encoded video data from the CPB and entropy decode the video data to reproduce syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310 and filtering unit 312 may generate decoded video material based on syntax elements extracted from the bit stream.

Typically, video decoder 300 reconstructs pictures on a block-by-block basis. Video decoder 300 may perform reconstruction operations on each block individually (where the block currently being reconstructed (ie, decoded) may be referred to as the "current block").

Entropy decoding unit 302 may entropy decode syntax elements that define quantized transform coefficients of a quantized transform coefficient block, and transform information such as quantization parameters (QPs) and/or transform mode indications. Inverse quantization unit 306 may use the QP associated with the quantized transform coefficient block to determine the degree of quantization and, likewise, determine the degree of inverse quantization for inverse quantization unit 306 to apply. For example, inverse quantization unit 306 may perform a bitwise left shift operation to inversely quantize the quantized transform coefficients. Inverse quantization unit 306 may thereby form a transform coefficient block including transform coefficients.

After inverse quantization unit 306 forms the block of transform coefficients, inverse transform processing unit 308 may apply one or more inverse transforms to the block of transform coefficients to produce a residual block associated with the current block. For example, the inverse transform processing unit 308 may apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotation transform, an inverse direction transform, or another inverse transform to the transform coefficient block.

In addition, the prediction processing unit 304 generates prediction blocks based on the prediction information syntax elements entropy-decoded by the entropy decoding unit 302 . For example, if the prediction information syntax element indicates that the current block is inter-predicted, motion compensation unit 316 may generate a prediction block. In this case, the prediction information syntax element may indicate the reference picture in DPB 314 from which the reference block was retrieved, and a motion vector identifying the position of the reference block in the reference picture relative to the current block in the current picture. Motion compensation unit 316 may generally perform inter-frame prediction processing in a manner substantially similar to that described with respect to motion compensation unit 224 (FIG. 2).

As another example, if the prediction information syntax element indicates that the current block is intra-predicted, the intra prediction unit 318 may generate the prediction block according to the intra prediction mode indicated by the prediction information syntax element. Again, intra-prediction unit 318 may generally perform intra-prediction processing in a manner substantially similar to that described with respect to intra-prediction unit 226 (FIG. 2). Intra-frame prediction unit 318 may retrieve adjacent sample data of the current block from DPB 314 .

Reconstruction unit 310 may reconstruct the current block using the prediction block and the residual block. For example, reconstruction unit 310 may add samples of the residual block to corresponding samples of the prediction block to reconstruct the current block.

Filtering unit 312 may perform one or more filtering operations on the reconstructed blocks. For example, filtering unit 312 may perform deblocking operations to reduce blocking artifacts along edges of reconstructed blocks. The operation of filter unit 312 may not necessarily be performed in all instances.

According to one or more examples described in this content, the filtering unit 312 may be configured to perform neural network-based filtering techniques. For example, the filtering unit 312 may perform neural network-based filtering techniques in addition to or instead of other filtering techniques such as ALF and SAO. In some examples, filtering unit 312 may perform the neural network-based filtering described in this context as part of in-loop filtering. Although not shown, in some examples, a filtering unit may be coupled to the output of DPB 314, and the output from the filtering unit may be decoded video. In some examples, a filter unit coupled to the output of DPB 314 may be configured to perform the example neural network-based filtering techniques described in this context as part of post-loop filtering.

Video decoder 300 may store the reconstructed blocks in DPB 314. For example, in instances where the operations of filtering unit 312 are not performed, reconstruction unit 310 may store the reconstructed blocks into DPB 314. In an example in which the operations of filtering unit 312 are performed, filtering unit 312 may store the filtered reconstructed blocks in DPB 314 . As mentioned above, DPB 314 may provide reference information to prediction processing unit 304, such as samples of the current picture for intra-frame prediction and previously decoded pictures for subsequent motion compensation. Additionally, video decoder 300 may output decoded pictures (eg, decoded video) from DPB 314 for subsequent presentation on a display device, such as display device 118 of FIG. 1 .

In this manner, video decoder 300 represents an example of a video decoding device that includes a memory configured to store video data, and one or more processing units implemented with circuitry, where the one or more processing units Configured to filter two or more color components of video material using a NN-based filter from a single neural network (NN) model, and to output after a single filtering procedure using the NN-based filter These two or more color components. In some examples, filtering includes: deciding how to apply a NN-based filter, whether to turn a NN-based filter on or off, how to create input data for a single NN model, and targeting two or more color components and Determine the values of all input elements in the same way.

In some examples, video decoder 300 may be configured to receive information indicating whether the NN-based filter is on or off for each of two or more color components. In such instances, filtering includes filtering based on whether a NN-based filter is on or off for each of the two or more color components.

For example, filtering unit 312 may receive a syntax element that defines a filtering mode for a neural network (NN) model for both the first color component and the second color component. Examples of syntax elements include the aforementioned first-level filtering mode syntax element, second-level filtering mode syntax element, first_chroma_mode_slice (for example, where first_chroma_mode_slice is non-zero and second_chroma_flag_slice is true) and chroma_mode_slice (for example, where first_chroma_flag_slice and/or second_chroma_flag_slice is true) real).

In some examples, the syntax element defines which parameter is selected from a plurality of parameters, and the defined parameter specifies the filtering mode. As an example, the parameter is a quantized parameter and the plurality of parameters is one of a predefined set or a signaled set. For example, a value of 1 for the first level filter mode syntax element indicates that the filter mode is defined by the first parameter in a predefined set or a signaled set, and a value of 2 for the first level filter mode syntax element indicates that A filter mode is defined by a second parameter in a predefined set or a signaled set, and a value of 3 for the first level filter mode syntax element indicates a third parameter in a predefined set or a signaled set to define the filter mode. The values of the second-level filter mode syntax elements first_chroma_mode_slice and chroma_mode_stice can similarly define the filter mode, as described in the example above.

Filtering unit 312 may apply an instance of the NN model to the first block of the first color component in a defined filtering mode to produce a first filtered block. Additionally, filtering unit 312 may apply the same instance of the NN model to the second block of the second color component in a defined filtering mode to produce a second filtered block. Filtering unit 312 may store the value for the CU based on the first filtered block. With NN-based filtering enabled for both the first and second color components, filtering unit 312 may store the value for the CU based on the first and second filtered blocks. As mentioned previously, CU is used as a term to describe a composite block that includes multiple different color components.

In this way, one syntax element defines the filtering mode for two or more color components, rather than having the syntax element define the filtering mode for each color component separately. For example, in execution of a NN model, filtering unit 312 may generate a first filtered block for a first color component and a second filtered block for a second color component in a defined filtering mode. Filtering unit 312 may store the value for the CU based on the first filtering block and, if filtering is enabled for the second color component, the value for the CU based on the first and second filtering blocks.

Storing the value of the CU based on the first filtered block and/or the second filtered block may mean that the samples of the CU stored in the DPB 314 are the first filtered block and the second filtered block, or if there is no filtered block for the first filtered block and/or the second filtered block. If filtering is enabled for the second color component, the samples of the stored CU are the first filtered block and the second block (unfiltered). As another example, the decoded video data output by DPB 314 may include a first filtered block and a second filtered block (if filtering is enabled for both), or if filtering is not enabled for the second color component, the DPB The decoded video data output by 314 may include a first filtered block and a second block (unfiltered).

As an example, the syntax element may be a syntax element applicable to a plurality of CUs. For example, the syntax element may be a first level filter mode syntax element that defines a filter mode for a plurality of CUs (eg, CUs within a slice). As another example, the syntax element may be a second-level filter mode syntax element that defines a filter mode for a subset of CUs. For example, video decoder 300 may receive a first syntax element applicable to a plurality of CUs, the first syntax element indicating that a second syntax element is to be parsed for a subset of CUs in the plurality of CUs (e.g., in the example above , the first level filtering mode syntax element is 4). In this example, the CU subset includes one or more CUs whose values are stored. Video decoder 300 may receive a second syntax element (e.g., a second level filtering mode syntax element) based on a first syntax element (e.g., a first level filtering mode syntax element) indicating that the second syntax element is to be parsed for a subset of CUs. ). That is, in the above example, when the value of the first-level syntax element is 4, the video decoder 300 may receive a second-level syntax element with a value between 0 and 3.

As mentioned before, whether to use the second filtered block (eg, generated according to NN-based filtering) may be set based on signaled parameters. For example, a syntax element received by video decoder 300 that defines a filtering mode for a NN model for both a first color component and a second color component may have a value indicating that NN-based filtering is enabled for the first color component. For example, the value of first_chroma_mode_slice can be non-zero. In this example, if a flag (e.g., second_chroma_flag_slice) indicates that NN-based filtering is enabled for the second color component, filtering unit 312 may apply the same instance of the NN model to the second block via the defined filtering mode, A second filtered block is generated based on the second block of second color components.

That is, the first_chroma_mode_slice value may define the filtering mode for the NN model, and if filtering is enabled for the second color component, the value of first_chroma_mode_slice defines the filtering mode for the NN model for the second color component. Thus, in this example, filtering unit 312 may store values for CUs in a defined filtering mode using the first and second filtered blocks generated by applying the same instance of the NN model.

However, in instances where second_chroma_flag_slice is false (which indicates that NN-based filtering is disabled for the second color component), filtering unit 312 may store the value for the CU based on the first filtered block and the second block of the second color component. That is, filtering unit 312 may discard the second filtered block of the second color component and instead store the first filtered block and the second block in DPB 314 (eg, using NN filtering).

The use of second_chroma_flag_slice is an example. In some examples, such as where the syntax defining a filtering mode for two color components is chroma_mode_slice, video decoder 300 may receive a flag indicating that NN-based filtering is enabled for the first color component (eg, receive first_chroma_flag_slice or second_chroma_flag_slice). In such instances, video decoder 300 may receive a syntax element (eg, chroma_mode_slice) based on a flag indicating that NN-based filtering is enabled for the first color component.

4 is a flowchart illustrating an example method for encoding a current block in accordance with the techniques of this disclosure. The current block may include the current CU. Although described with respect to the video transcoder 200 (Figs. 1 and 2), it should be understood that other devices may be configured to perform similar methods to that of Fig. 4.

In this example, video transcoder 200 initially predicts the current block (350). For example, video transcoder 200 may form a prediction block for the current block. Video transcoder 200 may then calculate a residual block for the current block (352). To calculate the residual block, video transcoder 200 may calculate the difference between the original uncoded block and the predicted block of the current block. Subsequently, the video transcoder 200 may transform the residual block and quantize the transform coefficients of the residual block (354). Next, video transcoder 200 may scan the quantized coefficients of the residual block (356). During or after scanning, video transcoder 200 may entropy encode the transform coefficients (358). For example, the video transcoder 200 may use CAVLC or CABAC to encode the transform coefficients. Video transcoder 200 may then output the entropy-encoded data of the block (360).

5 is a flowchart illustrating an example method for decoding a current block of video material in accordance with the techniques of this disclosure. The current block may include the current CU. Although described with respect to video decoder 300 (Figures 1 and 3), it should be understood that other devices may be configured to perform methods similar to Figure 5.

Video decoder 300 may receive entropy-encoded information for the current block (eg, entropy-encoded prediction information and entropy-encoded information for transform coefficients of a residual block corresponding to the current block) (370). Video decoder 300 may entropy decode the entropy-encoded data to determine prediction information for the current block and reproduce transform coefficients of the residual block (372). Video decoder 300 may predict the current block (374), for example using an intra or inter prediction mode as indicated by prediction information for the current block, to calculate a predicted block for the current block. Video decoder 300 may then inverse scan (376) the rendered transform coefficients to create a block of quantized transform coefficients. Video decoder 300 may then inversely quantize the transform coefficients and apply the inverse transform to the transform coefficients to produce a residual block (378). Video decoder 300 may finally decode the current block by combining the prediction block and the residual block (380).

In one or more examples, the video decoder 300 may also be configured to perform neural network-based filtering according to one or more examples described in this document. For example, filtering unit 312 may be configured to perform in-loop filtering using neural network-based filtering. In some examples, the filtering unit coupled to the output of DPB 314 may be configured to perform filtering using neural network-based filtering as described in this context as part of post-loop filtering.

Although FIG. 5 is described with respect to video decoder 300, in some examples, video transcoder 200 also includes a reconstruction process, and video transcoder 200 (eg, via inverse quantization unit 210, inverse transform processing unit 212, and reconstruction Unit 214) may be configured to perform the example techniques of Figure 5. In some examples, filtering unit 216 may be configured to perform the neural network-based filtering techniques described in this context. For example, filtering unit 216 may perform in-loop filtering as part of the reconstruction loop of video transcoder 200 .

9 is a flowchart illustrating an example method for processing video data according to the techniques of this disclosure. For ease of illustration, these examples are described with respect to video decoder 300, although video transcoder 200 may perform similar operations.

Video decoder 300 may receive a syntax element that defines a filtering pattern for a neural network (NN) model for both the first color component and the second color component (900). Examples of syntax elements include the first-level filtering mode syntax element, the second-level filtering mode syntax element as described above, first_chroma_mode_slice (e.g., where first_chroma_mode_slice is non-zero and second_chroma_flag_slice is true) and chroma_mode_slice (e.g., where first_chroma_flag_slice and/or second_chroma_flag_slice is true) ). In some examples, the syntax element defines which parameter is selected from a plurality of parameters, and the defined parameter specifies the filtering mode. As an example, the parameter is a quantized parameter and the plurality of parameters is one of a predefined set or a signaled set.

As mentioned above, the value of the first-level filter mode syntax element can be between 0 and 4. A value of 0 may indicate that NN-based filtering is disabled for the color components of the blocks within the slice (as an example). A value of 1, 2, or 3 may indicate a filtering mode for the NN model for both the first and second color components (e.g., defining whether to use the first, second, or third QP in the set, and by filter mode defined by the first, second, or third QP in the set). A value of 4 may indicate that the filtering mode for the NN model is defined at a lower level (e.g., at the CTU level or CU level) by the second-level filtering mode syntax element. A value of 0 for the second-level filter mode syntax element may indicate that NN-based filtering is disabled for the color component (as an example). A value of 1, 2, or 3 may indicate a filtering mode for the NN model for both the first and second color components (e.g., defining whether to use the first, second, or third QP in the set, and by filter mode defined by the first, second, or third QP in the set).

first_chroma_mode_slice is another example of a syntax element that defines a filtering mode for a neural network (NN) model for both the first color component and the second color component. For example, if first_chroma_mode_slice has a non-zero value and NN-based filtering is enabled for the second color component, then first_chroma_mode_slice defines the filtering mode for both the first and second color components.

chroma_mode_slice is another example of a syntax element that defines a filtering mode for a neural network (NN) model for both the first color component and the second color component. For example, where first_chroma_flag_slice and/or second_chroma_flag_slice are true, the value of chroma_mode_slice may define the filtering mode (eg, whether the first, second, or third QP value in the set is used to define the filtering mode).

Filtering unit 312 may apply an instance of the NN model to the first block of the first color component in the defined filtering mode to produce a first filtered block (902). For example, filtering unit 312 may apply an instance of the NN model to a first block of a first color component to generate a first residual value in a defined filtering mode, and generate a first residual value based on the first block and the first residual value. A first filtered block (eg, via summing a first residual value and a first filtered block).

In some examples, if NN-based filtering is enabled for the first color component, NN-based filtering is automatically enabled for the second color component. That is, in some instances, filtering of the second color component cannot be selectively omitted. Thus, in one or more instances, filtering unit 312 may apply the same instance of the NN model to the second block of the second color component in the defined filtering mode to produce a second filtered block (906). As mentioned previously, applying the same instance of a NN model may mean that the NN model receives as input both a first block of first color components and a second block of second color components (e.g., including a luminance component), and produces the first color A first filtered block of color components and a second filtered block of second color components are output.

That is, in some examples, a NN model for chrominance components (eg, a first color component and a second color component) may also receive a luma component as input, but output a first filtered block of the first color component and A second filtered block of a second color component. The NN model may not produce filtered luma blocks, but in some instances the NN model may produce filtered luma blocks.

In this case, filtering unit 312 may store the first filtered block and the second filtered block (908). For example, the sample values that filtering unit 312 stores in DPB 314 for the CU may be the values of the first filtered block and the second filtered block.

However, in some instances, utilizing the second filtered block may not always be true and may be optional. For example, in some examples, filtering unit 312 may determine whether to enable NN-based filtering for the second color component (904). For example, the filtering unit 312 may determine the value of first_chroma_flag_slice or second_chroma_flag_slice.

For example, the syntax element defining the filtering mode could be first_chroma_mode_slice. If the value of first_chroma_mode_slice is non-zero, the filtering unit 312 may determine the value of the flag (eg, second_chroma_flag_slice). That is, filtering unit 312 may receive a flag indicating that NN-based filtering is enabled for the second color component ("Yes" of 904). In this example, filtering unit 312 may apply the same instance of the NN model to the second block of the second color component in a defined filtering mode to produce a second filtered block (906), and store the first filtered block block and the second filtered block (908).

However, in some examples, filtering unit 312 may receive a flag indicating that NN-based filtering is disabled for the second color component (NO of 904 ). In this example, filtering unit 312 may store the first filtered block and the second block (eg, non-NN filtered block) (910).

Example techniques based on one or more examples described in this context are described below.

Clause 1A. A method of processing video data, the method comprising: filtering two or more color components of the video data using a NN-based filter from a single neural network (NN) model; and using the NN-based filter A single NN filter outputs the two or more color components after a single filtering procedure.

Clause 2A. The method according to clause 1A, wherein filtering includes: deciding how to apply the NN-based filter, whether to turn on or off the NN-based filter, how to establish input data for the single NN model, and for the two or two or more color components that determine the value of all input elements in the same way.

Clause 3A. The method according to clause 1A, further comprising: receiving or signaling information indicating whether the NN-based filter is turned on or off for each of the two or more color components, the filtering comprising: The NN-based filter is turned on or off for filtering for each of the two or more color components.

Clause 4A. The method according to any one of clauses 1A to 3A, further comprising: reconstructing sample values of the picture, wherein filtering includes: performing filtering on the two or more color components of the reconstructed sample values of the picture filter.

Clause 5A. The method according to clause 4A, wherein the filtering includes: in-loop filtering in a video transcoder or a video decoder.

Clause 6A. The method of clause 4A, wherein filtering includes: post-loop filtering in a video decoder.

Clause 7A. A device for processing video data, the device comprising: a memory configured to store the video data; and a processing circuit configured to perform the method according to any one of Clauses 1A-6A.

Clause 8A. The device described in clause 7A also includes a display configured to display the decoded video data.

Clause 9A. A device according to any of clauses 7A and 8A, wherein the device includes one or more of the following: a camera, a computer, a mobile device, a broadcast receiver device or a set-top box.

Clause 10A. Equipment according to any of clauses 7A-9A, wherein the equipment includes a video decoder.

Clause 11A. Equipment according to any of clauses 7A-10A, wherein the equipment includes a video transcoder.

Clause 12A. A computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to perform the method described in any one of Clauses 1A-6A.

Clause 13A. An apparatus for processing video data, the apparatus comprising: a unit for performing a method according to any one of clauses 1A-6A.

Clause 1B. A method of processing video data, the method comprising: receiving a syntax element defining a filtering pattern for a neural network (NN) model for both a first color component and a second color component; applying the instance of the NN model to the first block of the first color component in a defined filtering mode to produce a first filtered block; and storing the first filtered block for a coding unit (CU).

Clause 2B. The method of clause 1B, further comprising applying the same instance of the NN model to the second block of the second color component in the defined filtering mode to produce a second filtered block, wherein storing includes: The first filtered block and the second filtered block are stored for the CU.

Clause 3B. A method as in any one of clauses 1B and 2B, wherein receiving the syntax element includes receiving the syntax element applicable to a plurality of CUs, wherein the CU is one of the plurality of CUs.

Clause 4B. A method according to any one of clauses 1B-3B, wherein the syntax element is a second syntax element, the method further comprising: receiving a first syntax element applicable to a plurality of CUs, the first syntax element indicating that for A CU subset of the plurality of CUs parses the second syntax element, wherein the CU subset includes one or more CUs that include the CU, and wherein receiving the syntax element includes: indicating that the syntax element is targeted for the CU based on the first syntax element. subset to parse the second syntax element to receive the second syntax element.

Clause 5B. A method as in any of clauses 1B-4B, wherein the syntax element has a value indicating that NN-based filtering is enabled for the first color component.

Clause 6B. The method of clause 5B, further comprising: receiving a flag indicating that NN-based filtering is enabled for the second color component; and applying the same instance of the NN model to the second block via a defined filtering mode. , generating a second filtered block based on the second block of the second color component, wherein storing includes storing the first filtered block and the second filtered block for the CU.

Clause 7B. The method of clause 5B, further comprising receiving a flag indicating that NN-based filtering is disabled for the second color component, wherein storing includes storing, for the CU, a second filtered block of the first filtered block and the second color component. block.

Clause 8B. The method of any one of clauses 1B-7B, further comprising: receiving a flag indicating that NN-based filtering is enabled for the first color component, wherein receiving the syntax element includes indicating, based on the flag, that NN-based filtering is enabled for the first color component. The component enables NN-based filtering to receive this syntax element.

Clause 9B. A method according to any of clauses 1B-8B, wherein the syntax element defines which parameter is selected from the plurality of parameters, and wherein the defined parameter specifies the filtering mode.

Clause 10B. A method as described in clause 9B, wherein the parameter is a quantized parameter and the plurality of parameters is one of a predefined set or a signaled set.

Clause 11B. A method as in any of clauses 1B-10B, wherein the first color component is a first chrominance component and the second color component is a second chrominance component.

Clause 12B. A method according to any of clauses 1B-11B, wherein the instance of the NN mode is applied to the first block of the first color component in the defined filtering mode to produce the first filtered block , including: applying the instance of the NN model to the first block of the first color component in a defined filtering mode to produce a first residual value; and based on the first block and the first residual value difference, resulting in the first filtered block.

Clause 13B. A device for processing video data, the device includes: a memory configured to store the video data; and one or more processors implemented by circuitry coupled to the memory, configured to: receive A syntax element that defines a filtering mode for a neural network (NN) model for the first color component and the second color component; in the defined filtering mode, an instance of the NN model is applied to the a first block of color components to generate a first filtered block; and storing the first filtered block in the memory for a coding unit (CU).

Clause 14B. An apparatus as described in clause 13B, wherein the one or more processors are configured to apply the same instance of the NN model to the second block of the second color component in a defined filtering mode to produce a Two filtered blocks, wherein for storage, the one or more processors are configured to store the first filtered block and the second filtered block for the CU.

Clause 15B. An apparatus according to any of clauses 13B or 14B, wherein to receive the syntax element, the one or more processors are configured to: receive the syntax element applicable to a plurality of CUs, wherein the CU is the A CU among a plurality of CUs.

Clause 16B. An apparatus as in any one of clauses 13B-15B, wherein the syntax element is a second syntax element, and wherein the one or more processors are configured to receive a first syntax element applicable to a plurality of CUs. a syntax element, the first syntax element indicating that the second syntax element is to be parsed for a subset of CUs in the plurality of CUs, wherein the CU subset includes one or more CUs having the CU, wherein to receive the syntax element, The one or more processors are configured to receive the second syntax element based on the first syntax element indicating that the second syntax element is to be parsed for the subset of CUs.

Clause 17B. A device as in any one of clauses 13B-16B, wherein the syntax element has a value indicating that NN-based filtering is enabled for the first color component.

Clause 18B. The apparatus of clause 17B, wherein the one or more processors are configured to: receive a flag indicating that NN-based filtering is enabled for the second color component; and The same example applies to the second block, generating a second filtered block based on the second block of second color components, wherein for storage the one or more processors are configured to: store the first block for the CU filtered block and the second filtered block.

Clause 19B. The apparatus of clause 17B, wherein the one or more processors are configured to receive a flag indicating that NN-based filtering is disabled for the second color component, wherein for storage, the one or more processors are configured to : Stores the first filtered block and the second block of the second color component for the CU.

Clause 20B. An apparatus according to any of clauses 13B-19B, wherein the one or more processors are configured to: receive a flag indicating that NN-based filtering is enabled for the first color component, wherein to receive the syntax element, The one or more processors are configured to receive the syntax element based on the flag indicating that NN-based filtering is enabled for the first color component.

Clause 21B. An apparatus according to any one of clauses 13B-20B, wherein the syntax element defines which parameter is selected from the plurality of parameters, and wherein the defined parameter specifies the filtering mode.

Clause 22B. An apparatus as described in clause 21B, wherein the parameter is a quantized parameter and the plurality of parameters is one of a predefined set or a signaled set.

Clause 23B. An apparatus according to any of clauses 13B-22B, wherein the first color component is a first chrominance component and the second color component is a second chrominance component.

Clause 24B. Apparatus according to any of clauses 13B-23B, wherein in order to apply the instance of the NN mode to the first block of the first color component in the defined filtering mode to produce the first filtered block, the one or more processors configured to: apply the instance of the NN model to the first block of the first color component in a defined filtering mode to produce a first residual value; and Based on the first block and the first residual value, the first filtered block is generated.

Clause 25B. A computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to perform the following operations: receive a syntax element defined for a first color component and a second A filtering mode for a neural network (NN) model for both color components; in the defined filtering mode, an instance of the NN model is applied to the first block of the first color component to produce the first filtered block ; and storing the first filtered block for a coding unit (CU).

It should be appreciated that, depending on the example, certain actions or events of any technology described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all described actions or events may be required to implement the technology). required). Furthermore, in some instances, actions or events may be performed concurrently rather than sequentially, such as via multi-thread processing, interrupt processing, or multiple processors.

In one or more examples, the described functions may be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions may be stored on or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media or communication media, where communication media include devices that facilitate, for example, a computer program in accordance with a communications protocol. Any media transmitted from one place to another. In this manner, computer-readable media may generally correspond to: (1) non-transitory tangible computer-readable storage media; or (2) communications media such as signals or carrier waveforms. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to obtain instructions, code, and/or data structures for implementing the techniques described in this document. Computer program products may include computer-readable media.

By way of example, but not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk memory, magnetic disk memory or other magnetic storage devices, flash memory or Any other medium that can be used to store desired program code in the form of instructions or data structures that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, wireless, and microwave, Then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, wireless and microwave are included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waveforms, signals or other temporary media, but are directed to non-transitory tangible storage media. As used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray Discs, where disks usually copy data magnetically, while optical discs use lasers to optically copy data. Copy the data. The above combinations should also be included in the scope of protection of computer-readable media.

Instructions may be executed by one or more processors such as one or more DSPs, general purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or individual logic circuits. Thus, as used herein, the terms "processor" and "processing circuitry" may represent any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided in dedicated hardware and/or software modules configured to implement encoding and decoding, or incorporated into a combined transcoder. Additionally, these techniques may be fully implemented in one or more circuits or logic components.

The technology at issue may be implemented using a variety of devices or devices, including the use of a wireless handheld device, an integrated circuit (IC), or a set of ICs (e.g., a chip set). Various components, modules or units are described in this text to emphasize the functional aspects of devices configured to perform the disclosed technology, but do not necessarily need to be implemented by different hardware units. Instead, as mentioned above, the individual units may be combined in a transcoder hardware unit, or provided via a coordinated set of hardware units (including one or more processors as mentioned above) in conjunction with appropriate software and/or firmware .

Various examples are described. These and other examples fall within the scope of the appended claims.

100: Video encoding and decoding system 102: Source device 104:Video source 106:Memory 108:Output interface 110: Computer readable media 112:Storage device 114:File server 116:Destination device 118:Display device 120:Memory 122:Input interface 130: Modern hybrid video decoder 132: Enter video data 134:Summing unit 136: Transformation unit 138: Quantization unit 140: Entropy decoding unit 142: Inverse quantization unit 144: Inverse transformation unit 146:Summing unit 148: Loop filter unit 150: Decoded Picture Buffer (DPB) 152: In-frame prediction unit 154: Inter-frame prediction unit 156: Motion estimation unit 158: Output bit stream 200:Video transcoder 202: Mode selection unit 204: Residual generation unit 206: Transformation processing unit 208: Quantization unit 210: Inverse quantization unit 212: Inverse transformation processing unit 214: Reconstruction unit 216: Filter unit 218: Decoded Picture Buffer (DPB) 220: Entropy coding unit 222: Motion estimation unit 224: Motion compensation unit 226: In-frame prediction unit 230: Video data memory 300:Video decoder 302: Entropy decoding unit 304: Prediction processing unit 306: Inverse quantization unit 308: Inverse transformation processing unit 310: Reconstruction unit 312: Filter single 314: Decoded Picture Buffer (DPB) 316: Motion compensation unit 318: In-frame prediction unit 320: Decoded picture buffer (CPB) memory 350:block 352:Block 354:Block 356:block 358:Block 360:block 370:block 372:Block 374:Block 376:Block 378:Block 380:block 382:Block 700: Application Picture Group (GOP) 800: Reconstruction sampling 802: Residual sampling 804: Filtered sampling 900:block 902:Block 904:Block 906:block 908: Square 910:block

FIG. 1 is a block diagram illustrating an example video encoding and decoding system that may perform the techniques of this disclosure.

FIG. 2 is a block diagram illustrating an example video transcoder that may perform the techniques of this disclosure.

3 is a block diagram illustrating an example video decoder that may perform the techniques of this disclosure.

4 is a flowchart illustrating an example method for encoding a current block in accordance with the techniques of this disclosure.

FIG. 5 is a flowchart illustrating an example method for decoding a current block in accordance with the techniques of this disclosure.

FIG. 6 is a flowchart illustrating an example of a hybrid video coding framework similar to FIG. 2 .

7 is a conceptual diagram illustrating an example of a hierarchical prediction structure with a group of pictures (GOP) size equal to 16.

FIG. 8 is a conceptual diagram showing an example of a convolutional neural network (CNN).

FIG. 9 is a flowchart showing an example method of processing video data according to the technology of this case.

Domestic storage information (please note in order of storage institution, date and number) without Overseas storage information (please note in order of storage country, institution, date, and number) without

900:block

902:Block

904:Block

906:Block

908: Square

910:block

Claims (25)

A method for processing video data, the method includes the following steps: receiving a syntax element defining a filtering pattern for a neural network (NN) model for both a first color component and a second color component; In the defined filtering mode, applying an instance of the NN model to a first block of the first color component to produce a first filtered block; and The first filtered block is stored for a coding unit (CU). The method according to claim 1 also includes the following steps: applying the same instance of the NN model to a second block of the second color component in the defined filtering mode to produce a second filtered block, The storing includes the following steps: storing the first filtered block and the second filtered block for the CU. The method according to claim 1, wherein receiving the syntax element includes the following steps: receiving the syntax element applicable to a plurality of CUs, wherein the CU is one of the plurality of CUs. According to the method of claim 1, wherein the syntax element is a second syntax element, the method also includes the following steps: Receive a first syntax element applicable to a plurality of CUs, the first syntax element indicating that the second syntax element is to be parsed for a subset of CUs in the plurality of CUs, wherein the CU subset includes a CU that contains the CU. or multiple CUs, The receiving the syntax element includes the following steps: parsing the second syntax element for the CU subset based on the first syntax element indication to receive the second syntax element. The method of claim 1, wherein the syntax element has a value indicating that NN-based filtering is enabled for the first color component. The method according to claim 5 also includes the following steps: receiving a flag indicating that NN-based filtering is enabled for the second color component; and generating a second filtered block based on a second block of second color components by applying the same instance of the NN model to the second block in a defined filtering mode, The storing includes the following steps: storing the first filtered block and the second filtered block for the CU. The method according to claim 5 also includes the following steps: receiving a flag indicating that NN-based filtering is disabled for the second color component, The storing includes: storing the first filtered block and a second block of the second color component for the CU. The method according to claim 1 also includes the following steps: receiving a flag indicating that NN-based filtering is enabled for the first color component, wherein receiving the syntax element includes the following steps: receiving the syntax element based on the flag indicating that NN-based filtering is enabled for the first color component. The method according to claim 1, wherein the syntax element defines which parameter is selected from the plurality of parameters, and wherein the defined parameter specifies the filtering mode. The method of claim 9, wherein the parameter is a quantized parameter and the plurality of parameters is one of a predefined set or a signaled set. The method of claim 1, wherein the first color component is a first chrominance component and the second color component is a second chrominance component. The method of claim 1, wherein applying the instance of the NN mode to the first block of the first color component in a defined filtering mode to produce the first filtered block includes: applying the instance of the NN model to the first block of the first color component in a defined filtering mode to produce a first residual value; and Based on the first block and the first residual value, the first filtered block is generated. A device used to process video data. The device includes: memory configured to store the video data; and One or more processors coupled to the memory are implemented using circuitry configured to: receiving a syntax element defining a filtering pattern for a neural network (NN) model for both a first color component and a second color component; In the defined filtering mode, applying an instance of the NN model to a first block of the first color component to produce a first filtered block; and In the memory, the first filtered block is stored for a coding unit (CU). The device of claim 13, wherein the one or more processors are configured to: applying the same instance of the NN model to a second block of the second color component in the defined filtering mode to produce a second filtered block, For storage, the one or more processors are configured to store the first filtered block and the second filtered block for the CU. The apparatus of claim 13, wherein to receive the syntax element, the one or more processors are configured to: receive the syntax element applicable to a plurality of CUs, wherein the CU is one of the plurality of CUs. The apparatus of claim 13, wherein the syntax element is a second syntax element, and wherein the one or more processors are configured to: Receive a first syntax element applicable to a plurality of CUs, the first syntax element indicating that the second syntax element is to be parsed for a subset of CUs in the plurality of CUs, wherein the CU subset includes a CU that contains the CU. or multiple CUs, In order to receive the syntax element, the one or more processors are configured to: receive the second syntax element based on the first syntax element indicating that the second syntax element is parsed for the CU subset. The apparatus of claim 13, wherein the syntax element has a value indicating that NN-based filtering is enabled for the first color component. The device of claim 17, wherein the one or more processors are configured to: receiving a flag indicating that NN-based filtering is enabled for the second color component; and generating a second filtered block based on a second block of second color components by applying the same instance of the NN model to the second block in a defined filtering mode, Wherein for storage, the one or more processors are configured to store the first filtered block and the second filtered block for the CU. The device of claim 17, wherein the one or more processors are configured to: receiving a flag indicating that NN-based filtering is disabled for the second color component, wherein for storage, the one or more processors are configured to store the first filtered block and a second block of the second color component for the CU. The device of claim 13, wherein the one or more processors are configured to: receiving a flag indicating that NN-based filtering is enabled for the first color component, In order to receive the syntax element, the one or more processors are configured to: receive the syntax element based on the flag indicating that NN-based filtering is enabled for the first color component. Apparatus according to claim 13, wherein the syntax element defines which parameter is selected from the plurality of parameters, and wherein the defined parameter specifies the filtering mode. Apparatus according to claim 21, wherein the parameter is a quantized parameter and the plurality of parameters is one of a predefined set or a signaled set. The apparatus of claim 13, wherein the first color component is a first chrominance component and the second color component is a second chrominance component. The apparatus of claim 13, wherein in order to apply the instance of the NN mode to the first block of the first color component in a defined filtering mode to produce the first filtered block, the one or more processes The server is configured as: applying the instance of the NN model to the first block of the first color component in a defined filtering mode to produce a first residual value; and Based on the first block and the first residual value, the first filtered block is generated. A computer-readable storage medium on which instructions are stored that, when executed, cause one or more processors to: receiving a syntax element defining a filtering pattern for a neural network (NN) model for both a first color component and a second color component; In the defined filtering mode, applying an instance of the NN model to a first block of the first color component to produce a first filtered block; and The first filtered block is stored for a coding unit (CU).