Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/045—Combinations of networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T9/00—Image coding
  • G06T9/002—Image coding using neural networks
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/117—Filters, e.g. for pre-processing or post-processing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/119—Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/129—Scanning of coding units, e.g. zig-zag scan of transform coefficients or flexible macroblock ordering [FMO]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/13—Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/146—Data rate or code amount at the encoder output
  • H04N19/147—Data rate or code amount at the encoder output according to rate distortion criteria
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/189—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
  • H04N19/192—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding the adaptation method, adaptation tool or adaptation type being iterative or recursive
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
  • H04N19/82—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation involving filtering within a prediction loop
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
  • H04N19/86—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving reduction of coding artifacts, e.g. of blockiness
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20084—Artificial neural networks [ANN]

Definitions

• Digital video capabilities can be incorporated into a wide range of devi ces, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio telephones, so-called “smart phones,” video teleconferencing devices, video streaming devices, and the like.
• Digital video devices implement video coding techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265/High Efficiency Video Coding (HEVC), and extensions of such standards.
• the video devices may transmit, receive, encode, decode, and/or store digital video information more efficiently by implementing such video coding techniques.
• a computer-readable storage medium has stored thereon instructions that, when executed, cause a processor to: decode a picture of video data; code a value for a syntax element representing a neural network model to be used to filter a portion of the decoded picture, the value representing an index into a set of predefined neural network models, the index corresponding to the neural network model in the set of pre -defined neural network models; and filter the portion of the decoded picture using the neural network model corresponding to the index.
• a device for filtering decoded video data includes means for decoding a picture of video data; means for coding a value for a syntax element representing a neural network model to be used to filter a portion of the decoded picture, the value representing an index into a set of pre-defined neural network models, the index corresponding to the neural network model in the set of pre-defined neural network models; and means for filtering the portion of the decoded picture using the neural network model corresponding to the index.
• VVC Versatile Video Coding
• ITU-T H.266 which has been developed by the Joint Video Expert TEAM (JVET) of ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Motion Picture Experts Group (MPEG).
• JVET Joint Video Expert TEAM
• MPEG ISO/IEC Motion Picture Experts Group
• Version 1 of tire VVC specification referred to as “WC FDIS” hereinafter, is available from http://phenix.int- evry.fr/jvet/doc_end_user/documents/19_Teleconference/wgl l/JVET-S200 l-vl7.zip.
• W FDIS Motion Picture Experts Group
• Video source 104 of source device 102 may include a video capture device, such as a video camera, a video archive containing previously captured raw video, and/or a video feed interface to receive video from a video content provider.
• video source 104 may generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video.
• video encoder 200 encodes the captured, pre-captured, or computer-generated video data.
• Video encoder 200 may rearrange the pictures from the received order (sometimes referred to as “display order”) into a coding order for coding.
• Video encoder 200 may generate a bitstream including encoded video data.
• Source device 102 may then output the encoded video data via output interface 108 onto computer-readable medium 110 for reception and/or retrieval by, e.g., input interface 122 of destination device 116.
• Computer-readable medium 110 may represent any type of medium or device capable of transporting the encoded video data from source device 102 to destination device 116.
• computer-readable medium 110 represents a communication medium to enable source device 102 to transmit encoded video data directly to destination device 116 in real-time, e.g., via a radio frequency network or computer-based network.
• Output interface 108 may modulate a transmission signal including the encoded video data, and input interface 122 may demodulate the received transmission signal, according to a communication standard, such as a wireless communication protocol.
• the communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines.
• RF radio frequency
• source device 102 may output encoded data from output interface 108 to storage device 112.
• destination device 116 may access encoded data from storage device 112 via input interface 122.
• Storage device 112 may include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory-, or any other suitable digital storage media for storing encoded video data.
• source device 102 may output encoded video data to file server 114 or another intermediate storage device that may store the encoded video data generated by source device 102. Destination device 116 may access stored video data from file server 114 via streaming or download.
• video encoder 200 may generate the prediction block using one or more motion vectors.
• Video encoder 200 may generally perform a motion search to identify a reference block that closely matches the CU, e.g., in terms of differences between the CU and the reference block.
• Video encoder 200 may calculate a difference metric using a sum of absolute difference (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared differences (MSD), or other such difference calculations to determine whether a reference block closely matches the current CU.
• SAD sum of absolute difference
• SSD sum of squared differences
• MAD mean absolute difference
• MSD mean squared differences
• video encoder 200 may predict the current CU using uni -directional prediction or bi-directional prediction.
• Quantization generally refers to a process in which transform coefficients are quantized to possibly reduce the amount of data used to represent the transform coefficients, providing further compression.
• video encoder 200 may reduce the bit depth associated with some or all of the transform coefficients. For example, video encoder 200 may round an «-bit value down to an m ⁇ bit value during quantization, where n is greater than m. in some examples, to perform quantization, video encoder 200 may perform a bitwise right-shift of the value to be quantized.
• Block partitioning is used to divide the image into smaller blocks for operation of the prediction and transform processes.
• the early video coding standards used a fixed block size, typically 16x 16 samples.
• Recent standards, such as HE VC and VVC employ tree-based partitioning structures to provide flexible partitioning.
• Quantization aims to reduce the precision of an input value or a set of input values in order to decrease the amount of data needed to represent the values.
• quantization is typically applied to individual transformed residual samples, i.e., to transform coefficients, resulting in integer coefficient levels.
• the step size is derived from a so-called quantization parameter (QP) that controls the fidelity and bit rate.
• QP quantization parameter
• a larger step size lowers the bit rate but also deteriorates the quality, which e.g., results m video pictures exhibiting blocking artifacts and blurred details.
• Context-adaptive binary arithmetic coding is a form of entropy coding used in recent video codecs, e.g., AVC, HEVC, and VVC, due to its high efficiency.
• the NN-based filtering process may take the reconstructed samples as inputs, and the intermediate outputs are residual samples, which are added back to the input to refine the input samples.
• the NN filter may use all color components (e.g., Y, U, and V, or Y, Cb, and Cr, i.e., luminance, blue-hue chrominance, and red-hue chrominance) as input to exploit cross-component correlations.
• Different color components may share the same filters (including network structure and model parameters) or each color component may have its own specific filters.
• M can be any value that is pre-defined or signaled in bitstreams at a lower level (e.g,, slice header, picture header, CTU level, grid level, etc.) If M > 1, one of the M candidates is selected and signaled in the bitstream as a syntax element.
• each of the filter models may be associated with a QP value, and for each picture, video encoder 200 and video decoder 300 may derive a ‘model selection QP’ and select the models with the QP value that is closest to the ‘model selection QP’ for the current picture. In this case, no additional information needs to be signaled.
• video encoder 200 includes video data memory' 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, decoded picture buffer (DPB) 218, and entropy encoding unit 22.0.
• Any or all of video data memorj' 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, DPB 218, and entropy encoding unit 220 may be implemented in one or more processors or in processing circuitry.
• video decoder 300 may be implemented as one or more circuits or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA.
• video decoder 300 may include additional or alternative processors or processing circuitry to perform these and other functions.
• video decoder 300 reconstructs a picture on a block-by-block basis.
• Video decoder 300 may perform a reconstruction operation on each block individually ( w here the block currently being reconstructed, i.e., decoded, may be referred to as a “current block”).
• Entropy decoding unit 302 may entropy decode syntax elements defining quantized transform coefficients of a quantized transform coefficient block, as well as transform information, such as a quantization parameter (QP) and/or transform mode indication(s).
• Inverse quantization unit 306 may use the QP associated with the quantized transform coefficient block to determine a degree of quantization and, likewise, a degree of inverse quantization for inverse quantization unit 306 to apply.
• Inverse quantization unit 306 may, tor example, perform a bitwise left-shift operation to inverse quantize the quantized transform coefficients. Inverse quantization unit 306 may thereby form a transform coefficient block including transform coefficients.
• FIG. 7 represents an example of a device for filtering decoded video data including a memory' configured to store video data; and one or more processors implemented in circuitry and configured to: decode a picture of video data; code a value for a syntax element representing a neural network model to be used to filter a portion of the decoded picture, the value representing an index into a set of pre-defined neural network models, the index corresponding to the neural network model in the set of pre-defined neural network models; and filter the portion of the decoded picture using the neural network model corresponding to the index.
• FIG. 8 is a flowchart illustrating an example method for encoding a current block in accordance with the techniques of this disclosure.
• Tire current block may comprise a current CU.
• Video decoder 300 may receive entropy encoded data for the current block, such as entropy encoded prediction information and entropy encoded data for coefficients of a residual block corresponding to the current block (370). Video decoder 300 may entropy decode the entropy encoded data to determine prediction information for the current block, a neural network (NN) model for a portion of the picture including the current block, and to reproduce coefficients of the residual block (372). Video decoder 300 may predict the current block (374), e.g., using an intra- or inter-prediction mode as indicated by the prediction information for the current block, to calculate a prediction block for the current block.
• entropy encoded data for the current block such as entropy encoded prediction information and entropy encoded data for coefficients of a residual block corresponding to the current block (370).
• Video decoder 300 may entropy decode the entropy encoded data to determine prediction information for the current block, a neural network (NN) model for
• Clause 1 A method of filtering decoded video data, the method comprising: determining one or more neural network models to be used to filter a portion of a decoded picture of video data; and filtering the portion of the decoded picture using the one or more neural network models.
• Clause 37 Tire method of clause 23, wherein the value for the syntax element comprises a quantization parameter (QP) for the portion of the picture.
• QP quantization parameter
• Clause 38 The method of clause 23, further comprising encoding the picture prior to decoding the picture, wherein coding the value for the syntax element comprises encoding the value for the syntax element.
• Clause 78 The method of clause 77, further comprising coding a syntax element that jointly represents filtering using the neural network model for each of the color components of the decoded picture.
• Clause 81 The method of clause 80, further comprising determining the neural network model according to a rate -distortion computation.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Artificial Intelligence (AREA)
• Evolutionary Computation (AREA)
• Data Mining & Analysis (AREA)
• Computational Linguistics (AREA)
• Biophysics (AREA)
• Biomedical Technology (AREA)
• General Health & Medical Sciences (AREA)
• Molecular Biology (AREA)
• Computing Systems (AREA)
• General Engineering & Computer Science (AREA)
• Life Sciences & Earth Sciences (AREA)
• Mathematical Physics (AREA)
• Software Systems (AREA)
• Health & Medical Sciences (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)

Priority Applications (6)

Applications Claiming Priority (4)

Publications (1)

Family

ID=80821639

Family Applications (1)

Country Status (8)

Cited By (3)

Families Citing this family (29)

Citations (1)

Family Cites Families (14)

• 2021
  • 2021-09-23 US US17/448,658 patent/US11930215B2/en active Active
  • 2021-09-24 CN CN202180064854.4A patent/CN116349226A/zh active Pending
  • 2021-09-24 KR KR1020237009249A patent/KR20230079360A/ko active Pending
  • 2021-09-24 JP JP2023515167A patent/JP7795528B2/ja active Active
  • 2021-09-24 PH PH1/2023/550197A patent/PH12023550197A1/en unknown
  • 2021-09-24 EP EP21795101.1A patent/EP4222960A1/en active Pending
  • 2021-09-24 WO PCT/US2021/052008 patent/WO2022072245A1/en not_active Ceased
  • 2021-09-24 BR BR112023004814A patent/BR112023004814A2/pt unknown
• 2024
  • 2024-02-06 US US18/433,946 patent/US12356014B2/en active Active
• 2025
  • 2025-06-19 US US19/243,453 patent/US20250317604A1/en active Pending

Patent Citations (1)

Non-Patent Citations (3)

Also Published As

Similar Documents

Legal Events