WO2025258909A1

WO2025258909A1 - Method and system for performing content adaptive based in loop filtering in video codec

Info

Publication number: WO2025258909A1
Application number: PCT/KR2025/007639
Authority: WO
Inventors: Neeraj Jayant Gadgil; Anubhav Singh; Raj Narayana Gadde; Kwangpyo CHOI; Minwoo Park; Yinji Piao
Original assignee: Samsung Electronics Co Ltd
Current assignee: Samsung Electronics Co Ltd
Priority date: 2024-06-12
Filing date: 2025-06-04
Publication date: 2025-12-18
Anticipated expiration: 2026-12-12

Abstract

A method for performing ILF by a video decoder is provided. The method may include obtaining a bit-stream comprising of an encoded source video information; generating a reconstructed video frame corresponding to the encoded source video frame by using the received bit-stream; obtaining one or more domain-transfer functions for each pixel block in the reconstructed video frame from the received bit-stream; obtaining one or more inverse functions by using the obtained one or more domain-transfer functions; generating one or more domain-transferred pixel blocks by applying the obtained one or more domain-transfer functions to each pixel block; generating one or more updated domain-transferred pixel blocks based on inputting the generated one or more domain-transferred pixel blocks to a pre-defined AI model; generating an updated reconstructed video frame by applying the obtained one or more inverse functions on each updated domain-transferred pixel block.

Description

METHOD AND SYSTEM FOR PERFORMING CONTENT ADAPTIVE BASED IN LOOP FILTERING IN VIDEO CODEC

This application is based on and derives the benefit of Indian Provisional Application 202441045463 filed on 12th June 2024, the contents of which are incorporated herein by reference. Furthermore, the Complete Specification of the said Provisional Application No. 202441045463 have been successfully filed with the Indian Patent Office on 21st October 2024. Embodiments disclosed herein relate to video processing systems, and more particularly to methods and systems for performing content adaptive based pre-processing and post-processing for In-Loop Filtering (ILF) in video codec.

In typical video codecs, reconstructed video frames differ from the source frames often manifesting in visual artifacts, for example, ringing, blockiness, texture-loss, and so on. To mitigate this, a set of tools, together known as an "In-Loop Filtering" (ILF) module, is introduced to improve the quality of reconstruction. ILF is integrated within the coding loop to enhance the quality of the reference frames used for predicting subsequent frames, thereby improving both subjective and objective compression performance.

Recently, AI-based ILF tools have gained attention due to their potential to learn complex visual patterns and restore finer details lost during compression. These tools typically employ deep neural networks (NNs) trained on large datasets to perform enhancement on reconstructed frames. However, a significant challenge with such NN-based ILFs is their limited generalization capability. Since the network parameters are learned from specific training datasets, their performance often fails to remain consistent across varied video content, capture devices, resolutions, and application domains. Meanwhile, enhance or enhancement for the reconstructed block or reconstructed video frame may be to perform in-loop filtering.

In particular, a neural network trained on a narrowly defined dataset may exhibit data bias, resulting in suboptimal performance on out-of-distribution samples such as user-generated content (UGC), animated sequences, or screen content. This lack of robustness poses a critical barrier to practical deployment, where the NN model is expected to operate reliably across all types of video data, including but not limited to professionally graded films, consumer camera footage, screen captures, and synthetic computer-generated imagery.

In an embodiment of the disclosure, a method for performing In-Loop Filtering (ILF) by a video decoder is provided. The method may include obtaining a bit-stream comprising of an encoded source video information. The method may include generating a reconstructed video frame corresponding to the encoded source video frame by using the received bit-stream. The method may include obtaining one or more domain-transfer functions for each pixel block in the reconstructed video frame from the received bit-stream. The method may include obtaining one or more inverse functions by using the obtained one or more domain-transfer functions. The method may include generating one or more domain-transferred pixel blocks by applying the obtained one or more domain-transfer functions to each pixel block. The method may include generating one or more updated domain-transferred pixel blocks based on inputting the generated one or more domain-transferred pixel blocks to a pre-defined Artificial Intelligence (AI) model. The method may include generating an updated reconstructed video frame by applying the obtained one or more inverse functions on each updated domain-transferred pixel block.

In an embodiment of the disclosure, a method for performing a content adaptive Artificial Intelligence (AI) based In-Loop Filtering (ILF) by a video encoder is provided. The method may include generating a reconstructed video frame derived from a source video frame. The method may include computing one or more histograms representing pixel values of at least one of one or more color channels, and one or more frequency transform coefficients for each pixel block in the reconstructed video frame. The method may include obtaining a metadata comprising a histogram representing average pixel values of at least one of the one or more color channels, and one or more frequency transform coefficients of a content, where the content is used to train a pre-trained AI model of an ILF. The method may include comparing at least one of the computed one or more histograms, and the one or more frequency transform coefficients of each pixel block with the obtained metadata. The method may include determining one or more domain-transfer functions, and corresponding one or more inverse functions for each pixel block, based on a result of the comparing. The method may include generating one or more domain-transferred pixel blocks by applying the determined one or more domain-transfer functions to each pixel block. The method may include generating one or more updated domain-transferred pixel blocks based on inputting the generated one or more domain-transferred pixel blocks to the pre-defined AI model of the ILF. The method may include generating an updated reconstructed video frame by applying the corresponding one or more inverse functions on each updated domain-transferred pixel block.

In an embodiment of the disclosure, a method for transmitting a bit-stream generated for performing a content adaptive Artificial Intelligence (AI) based In-Loop Filtering (ILF) by a video encoder is provided. The method may include generating a reconstructed video frame derived from a source video frame. The method may include computing one or more histograms representing pixel values of at least one of one or more color channels, and one or more frequency transform coefficients for each pixel block in the reconstructed video frame. The method may include obtaining a metadata comprising a histogram representing average pixel values of at least one of the one or more color channels, and one or more frequency transform coefficients of a content, where the content is used to train a pre-trained AI model of an ILF. The method may include comparing at least one of the computed one or more histograms, and the one or more frequency transform coefficients of each pixel block with the obtained metadata. The method may include determining one or more domain-transfer functions, and corresponding one or more inverse functions for each pixel block, based on a result of the comparing. The method may include generating one or more domain-transferred pixel blocks by applying the determined one or more domain-transfer functions to each pixel block. The method may include generating one or more updated domain-transferred pixel blocks based on inputting the generated one or more domain-transferred pixel blocks to the pre-defined AI model of the ILF. The method may include generating an updated reconstructed video frame by applying the corresponding one or more inverse functions on each updated domain-transferred pixel block.

These and other aspects of the example embodiments herein will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following descriptions, while indicating example embodiments and numerous specific details thereof, are given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the example embodiments herein without departing from the spirit thereof, and the example embodiments herein include all such modifications.

Embodiments herein are illustrated in the accompanying drawings, throughout which like reference letters indicate corresponding parts in the various figures. The embodiments herein will be better understood from the following description with reference to the following illustratory drawings. Embodiments herein are illustrated by way of examples in the accompanying drawings.

Figure 1 is a diagram illustrating an existing Versatile　Video　Coding (VVC) video decoder, according to an embodiments of the disclosure.

Figure 2 is a diagram illustrating pre-processing and post-processing modules for an NN-filter in a video decoder, according to an embodiments of the disclosure.

Figure 3 illustrates a block diagram for performing In-Loop Filtering (ILF) according to an embodiments of the disclosure.

Figure 4 illustrates a method for performing a content adaptive AI based ILF by the video encoder, according to an embodiments of the disclosure.

Figure 5 illustrates a flowchart for performing the content adaptive AI based ILF by the video encoder, according to an embodiments of the disclosure.

Figure 6 illustrates a method for performing ILF by the video decoder, according to an embodiments of the disclosure.

Figure 7 illustrates a flowchart for performing the content adaptive AI based ILF by the video decoder, according to an embodiments of the disclosure.

Figures 8a and 8b are diagrams depicting images captured using a camera, along with the histogram of the respective images, according to an embodiments of the disclosure.

Figure 8c is a diagram depicting a Computer-Generated Imagery (CGI) content, according to an embodiments of the disclosure.

Figure 8d is a diagram of a User-generated content (UGC) using a handheld capture device, according to an embodiments of the disclosure.

Figure 9 is a diagram depicting Histogram Correction Functions (HCF), according to an embodiments of the disclosure.

Figure 10 is a diagram depicting a Spectral Adjustment Function (SAF), according to an embodiments of the disclosure.

The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein may be practiced and to further enable those of skill in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

For the purposes of interpreting this specification, the definitions (as defined herein) will apply and whenever appropriate the terms used in singular will also include the plural and vice versa. It is to be understood that the terminology used herein is for the purposes of describing particular embodiments only and is not intended to be limiting. The terms "comprising", "having" and "including" are to be construed as open-ended terms unless otherwise noted.

The words/phrases "exemplary", "example", "illustration", "in an instance", "and the like", "and so on", "etc.", "etcetera", "e.g.," , "i.e.," are merely used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the present subject matter described herein using the words/phrases "exemplary", "example", "illustration", "in an instance", "and the like", "and so on", "etc.", "etcetera", "e.g.," , "i.e.," is not necessarily to be construed as preferred or advantageous over other embodiments.

Embodiments herein may be described and illustrated in terms of blocks which carry out a described function or functions.　These blocks, which may be referred to herein as managers, units, modules, hardware components or the like, are physically implemented by analog and/or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits and the like, and may optionally be driven by a firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.　The circuits constituting a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware to perform some functions of the block and a processor to perform other functions of the block.　Each block of the embodiments may be physically separated into two or more interacting and discrete blocks without departing from the scope of the disclosure.　Likewise, the blocks of the embodiments may be physically combined into more complex blocks without departing from the scope of the disclosure.

It should be noted that elements in the drawings are illustrated for the purposes of this description and ease of understanding and may not have necessarily been drawn to scale. For example, the flowcharts/sequence diagrams illustrate the method in terms of the steps required for understanding of aspects of the embodiments as disclosed herein. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the present embodiments so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein. Furthermore, in terms of the system, one or more components/modules which comprise the system may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the present embodiments so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

The accompanying drawings are used to help easily understand various technical features and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. As such, the present disclosure should be construed to extend to any modifications, equivalents, and substitutes in addition to those which are particularly set out in the accompanying drawings and the corresponding description. Usage of words such as first, second, third etc., to describe components/elements/steps is for the purposes of this description and should not be construed as sequential ordering/placement/occurrence unless specified otherwise.

In an embodiment of the disclosure, a system and method to employ a set of pre-processing and post-processing methods for an Artificial Intelligence (AI)-based In-Loop Filtering (ILF) is provided.

In an embodiment of the disclosure, a system and method to provide a training data-bias and make AI model generalizable in a reliable application for a variety of content is provided.

In an embodiment of the disclosure, a system and method for using a pixel histogram, and frequency information to statistically match inference data with a trained data of the AI model is provided.

In an embodiment of the disclosure, a system and method for applying a histogram-correction mechanism to the data before employing a Neural Network (NN) processing, and later applying a frequency-component adjustment mechanism to the data before employing the NN processing is provided.

In an embodiment of the disclosure, a system and method for performing an inverse process of the above steps of applying the histogram-correction mechanism and the frequency-component adjustment mechanism as post-processing after the application of the AI model, to obtain signal back in the original codeword-domain is provided.

In an embodiment of the disclosure, the diagram of Figure 1 is an example structure of an In-loop Filtering module 110 used in modern video coding systems for enhancing the quality of reconstructed video frames.

In an embodiment of the disclosure, the In-loop Filtering module 110 may obtain the reconstructed video frames. The in-loop filtering module 110 may include at least one of LMCS (Luma Mapping with Chroma Scaling) module 112, DBF (Deblocking Filter) module 114, SAO (Sample Adaptive Offset) module 116, and ALF / CC-ALF (Adaptive Loop Filter / Cross-Component Adaptive Loop Filter) module 118.

In an embodiment of the disclosure, the LMCS module 112 may perform non-linear mapping to the luma and chroma components of the reconstructed video frame in order to improve perceptual uniformity and to enhance compression efficiency. The mapping may be based on block-wise linear functions or look-up tables that transform the pixel values before subsequent filtering is applied.

In an embodiment of the disclosure, the DBF module 114 may perform filtering operations across block boundaries to reduce artifacts introduced by block-based prediction and transform coding. Deblocking filtering of the DBF module 114 may involve edge detection and adaptive filtering based on quantization parameters and boundary strength indicators.

In an embodiment of the disclosure, the SAO module 116 may classify reconstructed pixels into categories based on local gradients or band patterns and apply category-specific offset values to refine the pixel intensities. The goal of SAO is to correct systematic biases and reduce visual artifacts such as banding and ringing.

In an embodiment of the disclosure, ALF / CC-ALF module 118 may apply adaptive filtering based on filter coefficients derived from reconstructed pixel blocks. The filtering may be performed on a block-wise basis using trained or adaptively selected filter sets. In the case of CC-ALF (Cross-Component ALF), the filtering of one component (e.g., chroma) may be based on information derived from another component (e.g., luma), thereby leveraging inter-component correlation for improved filtering accuracy.

Meanwhile, for the convenience of explanation, modules that can be included in the in-loop filtering module are described, but are not limited to the disclosed examples, and other modules may be further included, or at least one module disclosed in FIG. 1 may be omitted. In addition, although it is described as divided into components called 'module' according to the operation or step performed, at least one processor can perform an operation or a step according to at least one module.

Recently, many Artificial Intelligence (AI)-based ILF tools have been proposed by various video standardization (for example, MPEG, AOMedia, JVET, and so on) stakeholders to further improve the reconstructed video. Such tools augment or replace one or more ILF tools. Further, such tools may not perform well across multiple datasets. For implementing a pre-trained NN model "as-is" in a product, generalization becomes critical for reliability and performance.

For implementing AI tools in a product, it becomes critical that NN models may perform reasonably well for all types of video data such as for various content: film-grade professional, User-Generated Content (UGC), screen and computer-generated imagery, content captures using various devices, and for various applications.

Currently, Neural Networks (NN) tend to perform better on the data that has similar characteristics as that of the data used for training. If a new data is to be processed, the performance may be inferior or uncertain. NN designed for in-loop processing of video sequences may not perform similarly across multiple datasets. Since a specific dataset(s) is used for training the network parameters, that dataset may contain some data-bias. The currently proposed AI models for in-loop filter models use a set of pre-determined data. This can be a significant issue for generalizing models to any data. A fundamental remedy is needed to improve NN model generalization.

Hence, there is a need in the art for solutions which will overcome the above mentioned drawback(s), among others.

In an embodiment of the disclosure, systems and methods for employing pre-processing and post-processing methods for Artificial Intelligence (AI) model for content-adaptive generalization in In Loop Filtering (ILF) in video codec may be provided. Referring now to the drawings, and more particularly to FIGS. 2 through 10, where similar reference characters denote corresponding features consistently throughout the figures, there are shown embodiments.

Figure 2 is a diagram illustrating pre-processing and post-processing modules for an NN-filter in a video decoder, according to an embodiments of the disclosure. In an embodiment of the disclosure, Figure 2 depicts an overview of the proposed system for performing pre-process and post-process of a reconstructed video frame around an AI based Neural Network (NN) filter in a video codec. As illustrated, the NN-filter can be generalized to an unseen data in a better way by employing a set of pre-processing and post-processing modules. A new data can be transferred to another domain prior to application of the NN-based processing and transferred back to its original domain subsequently. The system applies a histogram-correction mechanism followed by a frequency-component adjustment to the video data before employing a NN processing. Thereafter, the system performs an inverse process of the above steps of applying the histogram-correction mechanism and the frequency-component adjustment mechanism as post-processing after the application of the AI model, to obtain signal back in the original codeword-domain.

In an embodiment of the disclosure, the electronic device may be a video decoder or a video encoder. The electronic device may perform in-loop filtering. The In-loop Filtering module 210 may obtain at least one of a pre-process module 212, a NN-filter module 214, and a post-process module 216.

In an embodiment of the disclosure, the In-loop Filtering module 210 may obtain the reconstructed video frames. The pre-process module 212 may obtain pixel blocks in the reconstructed video frames. And, the pre-process module 212 may perform pre-processing. For example, the pre-process module 212 may obtain one or more domain-transfer functions for pixel block in the reconstructed video frame. And, the pre-process module 212 may apply the obtained one or more domain-transfer functions to each pixel block. The pre-process module 212 may generate one or more domain-transferred pixel blocks by applying the one or more domain-transfer functions to each pixel block.

In an embodiment of the disclosure, the NN-filter module 214 may obtain the generated one or more domain-transferred pixel blocks as an input. And, the NN-filter module 214 may output or generate one or more updated domain-transferred pixel blocks as an output. Meanwhile, the NN-filter module 214 may be referred to as pre-defined Artificial Intelligence (AI) model, AI model, or pre-trained AI model.

In an embodiment of the disclosure, the post-process module 216 may perform post-processing. For example, the post-process module 216 may obtain the one or more updated domain-transferred pixel blocks. And, the post-process module 216 may apply the one or more inverse functions of the one or more domain-transfer functions on each updated domain-transferred pixel block. The post-process module 216 may generate an updated reconstructed video frame by applying the one or more inverse functions on each updated domain-transferred pixel block.

Meanwhile, updated reconstructed blocks, or an updated reconstructed frame, etc. may represent blocks or frames on which in-loop filtering has been performed.

Meanwhile, the in-loop filtering according to an embodiment of the disclosure will be described in more detail below.

Figure 3 illustrates a block diagram for performing In-Loop Filtering (ILF) of a video codec. The system comprises a video encoder 310, and a video decoder 320. The video encoder 310 further comprises a processor 312, a communication module 314, and a memory module 316. The processor 312 of the video encoder 310 further comprises an encoder ILF module 313.

Meanwhile, the operations performed in the encoder ILF module 313 may be performed by the processor 312.

In an embodiment of the disclosure, the encoder ILF module 313 may receive at least one source video as input for filtering. The encoder ILF module 313 may generate a reconstructed video frame derived from the source video frame. The encoder ILF module 313 may compute one or more histograms representing pixel values of at least one of one or more color channels, and one or more frequency transform coefficients for each pixel block in the reconstructed video frame. The histograms are computed separately for each color channel in the form of a one-dimensional Look-Up Table (1D-LUT). The histograms are computed jointly for N channels in the form of a single N-dimensional LUT.

In an embodiment of the disclosure, the color channels may include, but not limited to, a luminance channel, and at least one chrominance channel. The spectral components of the color channels are computed in the form of the frequency transform coefficients using an invertible mathematical transformation. The invertible mathematical transformation may include, but not limited to, a two-dimensional (2D) Discrete Cosine Transform (DCT), and a 2D Discrete Wavelet Transform (DWT). Meanwhile, a two-dimensional (2D) Discrete Cosine Transform (DCT) may be referred to as a two-dimensional Discrete Cosine Transform (DCT-II).

In an embodiment of the disclosure, the encoder ILF module 313 may retrieve or obtain a metadata where the metadata comprises a histogram representing average pixel values of at least one of the color channels, and the frequency transform coefficients of a content. The content is used to train a pre-trained AI model of the ILF. In an embodiment of the disclosure, the content of the metadata is pre-computed, based on data used in a training process of the AI model. The content of the metadata may include, but not limited to, a professionally-captured content, a user-generated content, a computer-generated content, and a screen content.

In an embodiment of the disclosure, the encoder ILF module 313 may compare at least one of the computed histograms, and the frequency transform coefficients of each pixel block with the retrieved metadata. The encoder ILF module 313 may determine one or more domain-transfer functions for each pixel block, based on a result of the comparison. The encoder ILF module 313 may further determine inverse functions corresponding to the determined domain-transfer functions for each pixel block, based on the result of the comparison (comparing). The domain-transfer functions, and their corresponding one or more inverse functions are determined using at least one of one or more histogram correction functions, and one or more spectral adjustment functions. In an embodiment of the disclosure, the domain-transfer functions are designed and signaled for at least one of a pixel-block, a video frame, a scene, and a sequence level using a Rate Distortion Optimization (RDO) method.

In an embodiment of the disclosure, the encoder ILF module 313 may process the computed histograms of each pixel block of the reconstructed video frame, and the retrieved metadata using a Cumulative Distribution Function (CDF) matching. The encoder ILF module 313 may derive the histogram correction functions, from the processed histograms of each pixel block of the reconstructed video frame, and the retrieved metadata.

In an embodiment of the disclosure, the encoder ILF module 313 may determine an amount of scaling and shifting needed for the spectral components of the reconstructed pixel-block. The encoder ILF module 313 may match the determined amount of scaling and shifting with the retrieved metadata. The encoder ILF module 313 may derive the spectral adjustment functions from the matched determined amount of scaling and shifting with the retrieved metadata.

In an embodiment of the disclosure, the encoder ILF module 313 may generate one or more domain-transferred pixel blocks by applying the determined domain-transfer functions to each pixel block. The encoder ILF module 313 may generate one or more updated domain-transferred pixel blocks by feeding the generated domain-transferred pixel blocks to the pre-defined AI model of the ILF. The encoder ILF module 313 may generate one or more updated domain-transferred pixel blocks based on inputting the generated domain-transferred pixel blocks to the pre-defined AI model of the ILF. The encoder ILF module 313 may generate an updated reconstructed video frame by applying the corresponding one or more inverse functions of the domain-transfer functions on each updated domain-transferred pixel block, as an output.

In an embodiment of the disclosure, the encoder ILF module 313 may generate a bit-stream, where the bit-stream comprises one or more encoded source video frames, and corresponding domain-transfer functions. The encoder ILF module 313 may transmit the generated bit-stream to the video decoder 320.

In an embodiment of the disclosure, the video decoder 320 further comprises a processor 322, a communication module 324, and a memory module 326. The processor 322 of the video decoder 320 further comprises a decoder ILF module 323. Meanwhile, the operations performed in the encoder ILF module 313 may be performed by the processor 312.

In an embodiment of the disclosure, the decoder ILF module 323 may receive or obtain a bit-stream from the encoder ILF module 313 of the video encoder 310. The bit-stream comprises an encoded source video information. The decoder ILF module 323 may generate a reconstructed video frame corresponding to the encoded source video frame using the received bit-stream.

In an embodiment of the disclosure, the decoder ILF module 323 may extract or obtain one or more domain-transfer functions for each pixel block in the reconstructed video frame from the received bit-stream. For example, the decoder ILF module 323 may obtain or determine based on the domain-transfer function information obtained from the received bit-stream. The decoder ILF module 323 may compute or obtain one or more inverse functions by using the extracted or obtained domain-transfer functions. The decoder ILF module 323 may generate one or more domain-transferred pixel blocks by applying the extracted or obtained domain-transfer functions to each pixel block. The domain-transferred pixel blocks are generated by performing a domain transfer of one or more pixel blocks of the reconstructed video frame using at least one of the histogram correction functions, and the spectral adjustment functions. The domain-transfer functions may include at least one of the histogram correction functions, and the spectral adjustment functions.

In an embodiment of the disclosure, the decoder ILF module 323 may generate one or more updated domain-transferred pixel blocks by feeding the generated domain-transferred pixel blocks to a pre-defined AI model. The decoder ILF module 323 may generate one or more updated domain-transferred pixel blocks based on inputting the generated one or more domain-transferred pixel blocks to a pre-defined AI model. The decoder ILF module 323 may generate an updated reconstructed video frame by applying the computed or obtained inverse functions of the domain-transfer functions on each updated domain-transferred pixel block.

In an embodiment of the disclosure, the one or more domain-transferred pixel blocks may be generated by performing a domain transfer of one or more pixel blocks in the reconstructed video frame based on at least one of one or more histogram correction functions, and one or more spectral adjustment functions. The one or more domain-transfer functions may be generated or determined based on the pre-determined AI model metadata (it may be referred to as a metadata)(e.g., in an offline fashion). The metadata may be pre-computed based on the data used in the training process of the AI model. And, the metadata may include comprising one or more histograms of a content, the one or more spectral components of a content used to train the pre-trained AI model. The one or more histogram correction functions may be obtained by processing one or more histograms of each pixel block of the reconstructed video frame, and a metadata comprising one or more histograms of a content used to train the pre-trained AI model, using a Cumulative Distribution Function (CDF) matching. The one or more spectral adjustment functions may be obtained by determining an amount of scaling and shifting for matching one or more spectral components of the reconstructed pixel-block with the metadata comprising one or more spectral components of a content used to train the pre-trained AI model.

In an embodiment of the disclosure, the one or more histograms of each pixel block of the reconstructed video frame may be computed separately for each color channel in the form of a one-dimensional Look-Up Table (1D-LUT) or for N channels in the form of a single N-dimensional LUT. The one or more spectral components of the reconstructed pixel-block may be computed for one or more color channel in the form of transform-coefficients, using an invertible mathematical transformation, wherein the invertible mathematical transformation comprises at least one of a two-dimensional discrete cosine transform (DCT-II) or discrete wavelet transform (DWT).

Meanwhile, the one or more inverse functions may be determined or obtained by one or more domain-transfer functions. And, the one or more domain-transfer function may be designed ensuring the existence of the corresponding one or more inverse functions.

In an embodiment of the disclosure, wherein the one or more domain-transfer functions may be signaled for each pixel block, frame, scene, or sequence level as determined by the video encoder using rate-distortion optimization (RDO).

In an embodiment of the disclosure, the processor 312, and the processor 322 may process and execute data of a plurality of modules of the video encoder 310, and the video decoder 320 respectively. The processor 312, and the processor 322 may be configured to execute instructions stored in the memory module 316, and the memory module 326 respectively. The processor 312, and the processor 322 may comprise one or more of microprocessors, circuits, and other hardware configured for processing. The processor 312, and the processor 322 may be at least one of a single processer, a plurality of processors, multiple homogeneous or heterogeneous cores, multiple Central Processing Units (CPUs) of different kinds, microcontrollers, special media, and other accelerators. The processor 312, and the processor 322 may be an application processor (AP), a graphics-only processing unit (such as a graphics processing unit (GPU), a visual processing unit (VPU)), and/or an Artificial Intelligence (AI)-dedicated processor (such as a neural processing unit (NPU)).

In an embodiment of the disclosure, the plurality of modules of the processor 312, and the processor 322 of the video encoder 310, and the video decoder 320 may communicate via the communication module 314, and the communication module 324 respectively. The communication module 314, and the communication module 324 may be in the form of either a wired network or a wireless communication network module. The wireless communication network may comprise, but not limited to, Global Positioning System (GPS), Global System for Mobile Communications (GSM), Wi-Fi, Bluetooth low energy, Near-field communication (NFC), and so on. The wireless communication may further comprise one or more of Bluetooth, ZigBee, a short-range wireless communication (such as Ultra-Wideband (UWB)), and a medium-range wireless communication (such as Wi-Fi) or a long-range wireless communication (such as 3G/4G/5G/6G and non-3GPP technologies or WiMAX), according to the usage environment.

In an embodiment of the disclosure, the memory module 316, and the memory module 326 may comprise one or more volatile and non-volatile memory components which are capable of storing data and instructions of the modules of the video encoder 310, and the video decoder 320 to be executed. Examples of the memory module 316, and the memory module 326 may be, but not limited to, NAND, embedded Multi Media Card (eMMC), Secure Digital (SD) cards, Universal Serial Bus (USB), Serial Advanced Technology Attachment (SATA), solid-state drive (SSD), and so on. The memory module 316, and the memory module 326 may also include one or more computer-readable storage media. Examples of non-volatile storage elements may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. In addition, the memory module 316, and the memory module 326 may, in some examples, be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term "non-transitory" should not be interpreted to mean that the memory module 316, and the memory module 326 is non-movable. In certain examples, a non-transitory storage medium may store data that can, over time, change (for example, in Random Access Memory (RAM) or cache).

Figure 3 shows example modules of the video encoder 310, and the video decoder 320, but it is to be understood that other embodiments are not limited thereon. In other embodiments, the video encoder 310, and the video decoder 320 may include less or more number of modules. Further, the labels or names of the modules are used only for illustrative purpose and does not limit the scope of the invention. One or more modules may be combined together to perform same or substantially similar function in the video encoder 310, and the video decoder 320 respectively.

Figure 4 illustrates a method for performing a content adaptive AI based ILF by the video encoder, according to an embodiments of the disclosure. The video encoder may generate a reconstructed video frame from an original source video frame, as depicted in step 402. The video encoder may compute one or more histograms representing pixel values of at least one of one or more color channels, and one or more frequency transform coefficients for each pixel block in the reconstructed video frame, as depicted in step 404.

In an embodiment of the disclosure, the video encoder may retrieve a metadata comprising a histogram representing average pixel values of at least one of the color channels, and the frequency transform coefficients of a content, as depicted in step 406. The content is used to train a pre-trained AI model of an ILF. The video encoder may compare at least one of the computed histograms, and the frequency transform coefficients of each pixel block with the retrieved metadata, as depicted in step 408.

In an embodiment of the disclosure, the video encoder may determine one or more domain-transfer functions, and their corresponding inverse functions for each pixel block, based on a result of the comparison, as depicted in step 410. For example, the video encoder or video decoder may determine domain-transfer information for one or more domain-transfer function. The domain transfer information may include histogram correction information for one or more histogram correction function or spectral adjustment information for one or more spectral adjustment functions. The video encoder may generate one or more domain-transferred pixel blocks by applying the determined domain-transfer functions to each pixel block, as depicted in step 412.

In an embodiment of the disclosure, at step 414, the video encoder may generate one or more updated domain-transferred pixel blocks by feeding or inputting the generated or obtained domain-transferred pixel blocks to the pre-defined AI model of the ILF. At step 416, the video encoder may generate an updated reconstructed video frame by applying the corresponding inverse functions of the domain-transfer functions on each updated domain-transferred pixel block, as an output.

In an embodiment of the disclosure, the video encoder may generate one or more updated domain-transferred pixel blocks by feeding or inputting the generated domain-transferred pixel blocks to the pre-defined AI model of the ILF, as depicted in step 414. The video encoder may generate an updated reconstructed video frame by applying the corresponding inverse functions of the domain-transfer functions on each updated domain-transferred pixel block, as an output, as depicted in step 416.

In an embodiment of the disclosure, the various actions in method 400 may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some actions listed in Figure 4 may be omitted.

In an embodiment of the disclosure, the flowchart 500 depicts the pre-processing and post-processing at the video encoder 310. The encoder ILF module 313 of the video encoder 310 may receive a reconstructed video frame derived from a source video frame. The encoder ILF module 313 may collect luma/chroma histogram and frequency-domain statistics for each pixel block in the reconstructed video frame, as depicted in step 502. The encoder ILF module 313 may retrieve a pre-computed metadata comprising a histogram representing average pixel values of at least one of the color channels, and the frequency transform coefficients of a content for a Neural Network (NN) filter , as depicted in step 504.

In an embodiment of the disclosure, domain-transfer functions may be determined, by the encoder ILF module 313, using HCF (The Histogram Correction Function) and SAF (The Spectral Adjustment Functions), as depicted in step 506, based on the retrieved metadata. The encoder ILF module 313 further may generate a bit-stream using the HCF and SAF. The encoder ILF module 313 may compute inverse domain-transfer functions using the generated bit-stream, as depicted in step 514.

In an embodiment of the disclosure, the encoder ILF module 313 performs domain-transfer of each pixel block using HCF and SAF, as depicted in step 508. Later, the encoder ILF module 313 may apply the NN filter to the pixel block, as depicted in step 510, after performing domain-transfer. Later, the encoder ILF module 313 performs inverse domain-transfer of each pixel block, as depicted in step 512, using the computed inverse

domain-transfer functions at step 514. Thereafter, an updated reconstructed video frame is generated by applying the inverse functions.

In an embodiment of the disclosure, the various actions in method 500 may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some actions listed in Figure 5 may be omitted.

Figure 6 illustrates a method 600 for performing ILF by the video decoder, according to an embodiments of the disclosure. The video decoder may receive a bit-stream comprising of an encoded source video information, as depicted in step 602. The video decoder may generate a reconstructed video frame corresponding to the encoded source video frame using the received bit-stream, as depicted in step 604. The video decoder may extract one or more domain-transfer functions for each pixel block in the reconstructed video frame from the received bit-stream, as depicted in step 606. For example, the video encoder or video decoder may determine domain-transfer information for one or more domain-transfer function. The domain transfer information may include histogram correction information for one or more histogram correction function or spectral adjustment information for one or more spectral adjustment functions.

In an embodiment of the disclosure, the video decoder may compute one or more inverse functions using the extracted domain-transfer functions, as depicted in step 608. For example, the video encoder or video decoder may obtain the inverse functions corresponding to the domain-transfer functions by determining the inverse functions by pre-storing the inverse functions corresponding to the domain-transfer functions in the memory. However, without being limited to the disclosed example, the video decoder or video encoder can also be obtained by computing the inverse functions corresponding to the domain-transfer functions. The video encoder or video decoder may obtain the inverse information for inverse functions corresponding to the domain-transfer function. The video decoder may generate one or more domain-transferred pixel blocks by applying the extracted domain-transfer functions to each pixel block, as depicted in step 610. The video encoder or video decoder may generate one or more domain-transferred pixel blocks based on the domain-transfer information including at least one of histogram correction information, or spectral adjustment information. The video decoder may generate one or more updated domain-transferred pixel blocks by feeding the generated domain-transferred pixel blocks to a pre-defined AI model, as depicted in step 612. The video encoder or video decoder may generate one or more updated reconstructed blocks or an updated reconstructed video frame based on the inverse information. The video decoder may generate an updated reconstructed video frame by applying the computed inverse functions of the domain-transfer functions on each updated domain-transferred pixel block, as depicted in step 614.

Meanwhile, although omitted for convenience of explanation, operations in a video decoder may be performed identically or similarly in a video encoder, and a video decoder and a video encoder may be included in a single electronic device.

In an embodiment of the disclosure, the various actions in method 600 may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some actions listed in Figure 6 may be omitted.

Figure 7 illustrates a flowchart 700 for performing the content adaptive AI based ILF by the video decoder, according to an embodiments of the disclosure. The flowchart 700 depicts the pre- processing and post-processing at the video decoder 320. The decoder ILF module 323 of the video decoder 320 may receive a bit-stream comprising of an encoded source video information from the encoder ILF module 313 of the video encoder 310. The decoder ILF module 323 may generate a reconstructed video frame corresponding to the encoded source video frame using the received bit-stream, and perform domain-transfer using HCF (The Histogram Correction Functions) and SAF (The Spectral Adjustment Functions), as depicted in step 702.

In an embodiment of the disclosure, the decoder ILF module 323 may apply NN filter to the pixel block of the reconstructed video, as depicted in step 704, after performing domain-transfer. The decoder ILF module 323 may compute inverse domain-transfer functions using the generated bit-stream, as depicted in step 708. Later, the decoder ILF module 323 performs inverse domain-transfer of each pixel block, as depicted in step 706, using the computed inverse domain-transfer functions at step 708.

In an embodiment of the disclosure, the various actions in method 700 may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some actions listed in Figure 7 may be omitted.

In an embodiment of the disclosure, the proposed methods 400, 600 may include the pre-determined AI-model metadata, which are pre-computed i.e., in an offline fashion, based on the data used in the training process of the AI-model.

In an embodiment of the disclosure, the Histogram Correction Functions (HCF) may be derived by processing the histograms of the reconstructed pixel-block and the histograms received from the AI-model metadata using a method such as cumulative distribution function (CDF) matching.

In an embodiment of the disclosure, the Spectral Adjustment Functions (SAF) may be derived by determining the amount of scaling and shifting needed for the spectral components of the reconstructed pixel-block to match the spectral components received from the AI-model metadata.

In an embodiment of the disclosure, the domain-transfer functions may be designed ensuring the existence of their respective inverse functions. The domain-transfer functions may be designed and signaled for each pixel-block, frame, scene or sequence-level as determined by the video encoder 310 using a method known as Rate-Distortion Optimization (RDO).

In an embodiment of the disclosure, the proposed methods 400, 600 may determine domain-transfer functions for better generalization of AI-based in loop filter. Application of domain-transfer functions as a pre-processing and post-processing method for AI models at the video decoder 320.

In an embodiment of the disclosure, the proposed methods 400, 600 improve the performance and reliability of the AI model employed as in-loop processing tool. The methods 400, 600 are adaptive at a pixel-block level to the content that the pixels represent, without the need to train separate models or re-training existing model with a data of similar characteristics.

Figures 8a and 8b are diagrams depicting the images captured using a camera, along with the histogram of the respective images. Figure 8a is a 1088p image, and Figure 8b is a 544p image, each of which a 128x128 patch and its histogram is shown as an example. Due to the nature of the content, histogram (probability density function) is also quite different.

Figure 8c is a diagram depicting a Computer-Generated Imagery (CGI) content, according to an embodiments of the disclosure. Here, the histogram is considerably different than that of professional shooting.

Figure 8d is a diagram of a User-generated content (UGC) using a handheld capture device, according to an embodiments of the disclosure. In all these examples, a variety of content represented by their varied histograms is shown. If the AI model is agnostic to these input characteristics (statistics), the AI model may not perform well, leading to a sub-optimal AI performance. With the help of proposed pre-processing and post-processing functions, each data may be matched to the one that is used for AI-model training, such that the AI-model may be implemented for broad use-cases.

Figure 9 is a diagram depicting the Histogram Correction Functions (HCF), according to an embodiments of the disclosure. A pre-computed single-color channel (luminance or chrominance) histogram 930 is shown. The pre-computed single-color channel histogram may include a histogram of a pre-computed metadata 910 corresponding to training data used in the AI model (or NN filter) , and a histogram of reconstructed pixel-block in the reconstructed frame 920.

In an embodiment of the disclosure, a single-channel cumulative Distribution Function (CDF) corresponding to each histogram 940 may be computed. A CDF-matched Look-Up Table (LUT) may be then constructed using two CDFs. The resulting curve 950 may be used as HCF for a channel of the given pixel block during the pre-processing operation i.e., before the NN filter is applied.

In an embodiment of the disclosure, the CDF corresponding to each histogram 940 may be computed by the equation , but is not limited to it. The values of the resulting curve 950 may be computed by the equation , but is not limited to it.

In an embodiment of the disclosure, an inverse-mapped curve 960 may be computed from the resulting curve 950. The inverse-mapped curve 960 may be applied as a post-processing to the NN filter. In this case, a necessary condition to satisfy invertible function is monotonically non-decreasing nature of the CDF curves.

Figure 10 is a diagram depicting the Spectral Adjustment Function (SAF), according to an embodiments of the disclosure.

In an embodiment of the disclosure, a pre-computed frequency spectrum 1030, for example 2D DCT, corresponding to training data (e.g., metadata comprising one or more spectral components of a content 1010) used in the AI model (or NN filter) is shown. A pixel-block from the picture may be transformed into 2D DCT spectrum 1040. The 2D DCT spectrum is represented using grayscale from low to high values.

In an embodiment of the disclosure, since the spectra of the data used for training the AI model and the data it is employed on i.e., the pixel block are different, an SAF 1050 may be designed such that the AI model may convert the transform coefficients into another domain as pre-processing. The function may use multiply and add (offset) type of mathematical form for each coefficient.

In an embodiment of the disclosure, an inverse SAF 1060 may be designed, to obtain the data in the same domain during the post-processing. For computational ease and saving bit-stream parameters (to improve compression ratio), a set of separable (vertical and horizontal) SAFs 1070 may be designed and employed

In an embodiment of the disclosure, the proposed methods 400, 600 provide improved Bjontegaard Delta Rate (BD-Rate) performance. The compression performance of AI model, after using pre-processing and post-processing, is improved for a variety of content which is highly desirable for practical applications and deployment. The proposed methods 400, 600 facilitate device/product integration. The proposed methods 400, 600 are highly important in terms of improving performance and reliability of the AI model, and key to widespread adoption of AI-based content enhancement. Thus, the proposed methods 400, 600 use spatial statistic and frequency-domain properties to compare the inference data and training data, attempting to minimize data-bias while improving the model performance and reliability.

In an embodiment of the disclosure, the embodiments disclosed of the disclosure can be implemented through at least one software program running on at least one hardware device. The elements include blocks which can be at least one of a hardware device, or a combination of hardware device and software module.

The embodiments disclosed of the disclosure describe methods 400, 600 and systems for performing content adaptive based pre-processing and post-processing for ILF in video codec. Therefore, it is understood that the scope of the protection is extended to such a program and in addition to a computer readable means having a message therein, such computer readable storage means contain program code means for implementation of one or more steps of the method, when the program runs on a server or mobile device or any suitable programmable device. The method is implemented in at least one embodiment through or together with a software program written in e.g., Very high speed integrated circuit Hardware Description Language (VHDL) another programming language, or implemented by one or more VHDL or several software modules being executed on at least one hardware device. The hardware device can be any kind of portable device that can be programmed. The device may also include means which could be e.g., hardware means like e.g., an ASIC, or a combination of hardware and software means, e.g., an ASIC and an FPGA, or at least one microprocessor and at least one memory with software modules located therein. The method embodiments described herein could be implemented partly in hardware and partly in software. Alternatively, the invention may be implemented on different hardware devices, e.g., using a plurality of CPUs.

In an embodiment of the disclosure, method (600) for performing In-Loop Filtering (ILF) of a video codec by a video decoder (320) is provided. The method may include receiving a bit-stream comprising of an encoded source video information. The method may include generating a reconstructed video frame corresponding to the encoded source video frame using the received bit-stream. The method may include extracting one or more domain-transfer functions for each pixel block in the reconstructed video frame from the received bit-stream. The method may include computing one or more inverse functions using the extracted one or more domain-transfer functions. The method may include generating one or more domain-transferred pixel blocks by applying the extracted one or more domain-transfer functions to each pixel block. The method may include generating one or more updated domain-transferred pixel blocks by feeding the generated one or more domain-transferred pixel blocks to a pre-defined Artificial Intelligence (AI) model. The method may include generating an updated reconstructed video frame by applying the computed one or more inverse functions of the one or more domain-transfer functions on each updated domain-transferred pixel block.

In an embodiment of the disclosure, the one or more domain-transferred pixel blocks may be generated by performing a domain transfer of one or more pixel blocks of the reconstructed video frame using at least one of one or more histogram correction functions, and one or more spectral adjustment functions.

In an embodiment of the disclosure, the method may include processing one or more histograms of each pixel block of the reconstructed video frame, and a metadata comprising one or more histograms of a content, where the content is used to train the pre-trained AI model, using a Cumulative Distribution Function (CDF) matching. The method may include deriving the one or more histogram correction functions, from the processed one or more histograms of each pixel block of the reconstructed video frame, and the metadata.

In an embodiment of the disclosure, the method may include determining an amount of scaling and shifting needed for one or more spectral components of the reconstructed pixel-block. The method may include matching the determined amount of scaling and shifting with the metadata comprising the one or more histograms of the content. The method may include deriving the one or more spectral adjustment functions from the matched determined amount of scaling and shifting with the retrieved metadata.

In an embodiment of the disclosure, a video decoder (320) may include a processor (322), and a memory module (326). The processor (322) is coupled with the memory module (326), and configured to receive a bit-stream comprising of an encoded source video information. The processor (322) is configured to generate a reconstructed video frame corresponding to the encoded source video frame using the received bit-stream. The processor (322) is configured to extract one or more domain-transfer functions for each pixel block in the reconstructed video frame from the received bit-stream. The processor (322) is configured to compute one or more inverse functions using the extracted one or more domain-transfer functions. The processor (322) is configured to generate one or more domain-transferred pixel blocks by applying the extracted one or more domain-transfer functions to each pixel block. The processor (322) is configured to generate one or more updated domain-transferred pixel blocks by feeding the generated one or more domain-transferred pixel blocks to a pre-defined Artificial Intelligence (AI) model. The processor (322) is configured to generate an updated reconstructed video frame by applying the computed one or more inverse functions of the one or more domain-transfer functions on each updated domain-transferred pixel block.

In an embodiment of the disclosure, a method (400) for performing a content adaptive Artificial Intelligence (AI) based In-Loop Filtering (ILF) for a video codec by a video encoder (310) is provided. The method may include generating a reconstructed video frame derived from a source video frame. The method may include computing one or more histograms representing pixel values of at least one of one or more color channels, and one or more frequency transform coefficients for each pixel block in the reconstructed video frame. The method may include retrieving a metadata comprising a histogram representing average pixel values of at least one of the one or more color channels, and the one or more frequency transform coefficients of a content, where the content is used to train a pre-trained AI model of an ILF. The method may include comparing at least one of the computed one or more histograms, and the one or more frequency transform coefficients of each pixel block with the retrieved metadata. The method may include determining one or more domain-transfer functions, and their corresponding inverse functions for each pixel block, based on a result of the comparison. The method may include generating one or more domain-transferred pixel blocks by applying the determined one or more domain-transfer functions to each pixel block. The method may include generating one or more updated domain-transferred pixel blocks by feeding the generated one or more domain-transferred pixel blocks to the pre-defined AI model of the ILF. The method may include generating an updated reconstructed video frame by applying the corresponding inverse functions of the one or more domain-transfer functions on each updated domain-transferred pixel block, as an output.

In an embodiment of the disclosure, the method may include generating a bit-stream comprising of one or more encoded source video frames, and corresponding one or more domain-transfer functions. The method may include transmitting the generated bit-stream to a video decoder (320).

In an embodiment of the disclosure, the one or more histograms may be computed separately for each color channel in the form of a one-dimensional Look-Up Table (1D-LUT), wherein the one or more histograms are computed jointly for N channels in the form of a single N-dimensional LUT.

In an embodiment of the disclosure, the one or more color channels may include at least one of a luminance channel, and a chrominance channel.

In an embodiment of the disclosure, the content of the metadata is pre-computed, based on data used in a training process of the AI model, wherein the content of the metadata comprises at least one of a professionally-captured content, a user-generated content, a computer-generated content, and a screen content.

In an embodiment of the disclosure, the one or more spectral components of the one or more color channels may be computed in the form of the one or more frequency transform coefficients using an invertible mathematical transformation, wherein the invertible mathematical transformation comprises at least one of a two-dimensional (2D) Discrete Cosine Transform (DCT), and a 2D Discrete Wavelet Transform (DWT).

In an embodiment of the disclosure, the one or more domain-transfer functions, and their corresponding inverse functions may be determined using at least one of one or more histogram correction functions, and one or more spectral adjustment functions.

In an embodiment of the disclosure, the method may include processing the computed one or more histograms of each pixel block of the reconstructed video frame, and the retrieved metadata using a Cumulative Distribution Function (CDF) matching. The method may include deriving the one or more histogram correction functions, from the processed one or more histograms of each pixel block of the reconstructed video frame, and the retrieved metadata.

In an embodiment of the disclosure, the method may include determining an amount of scaling and shifting needed for the one or more spectral components of the reconstructed pixel-block. The method may include matching the determined amount of scaling and shifting with the retrieved metadata. The method may include deriving the one or more spectral adjustment functions from the matched determined amount of scaling and shifting with the retrieved metadata.

In an embodiment of the disclosure, the one or more domain-transfer functions are designed and signaled for at least one of a pixel-block, a video frame, a scene, and a sequence level using a Rate Distortion Optimization (RDO) method.

In an embodiment of the disclosure, a video encoder (310) may include a processor (312), and a memory module (316). The processor (312) may be coupled with the memory module (316), and configured to generate a reconstructed video frame derived from a source video frame. The processor (312) may be configured to compute one or more histograms representing pixel values of at least one of one or more color channels, and one or more frequency transform coefficients for each pixel block in the reconstructed video frame. The processor (312) may be configured to retrieve a metadata comprising a histogram representing average pixel values of at least one of the one or more color channels, and the one or more frequency transform coefficients of a content, where the content is used to train a pre-trained AI model of an In-Loop Filtering (ILF). The processor (312) may be configured to compare at least one of the computed one or more histograms, and the one or more frequency transform coefficients of each pixel block with the retrieved metadata. The processor (312) may be configured to determine one or more domain-transfer functions, and their corresponding inverse functions for each pixel block, based on a result of the comparison. The processor (312) may be configured to generate one or more domain-transferred pixel blocks by applying the determined one or more domain-transfer functions to each pixel block. The processor (312) may be configured to generate one or more updated domain-transferred pixel blocks by feeding the generated one or more domain-transferred pixel blocks to the pre-defined AI model of the ILF. The processor (312) may be configured to generate an updated reconstructed video frame by applying the corresponding inverse functions of the one or more domain-transfer functions on each updated domain-transferred pixel block, as an output.

In an embodiment of the disclosure, a method (600) for performing In-Loop Filtering (ILF) by a video decoder (320), is provided. The method may include obtaining a bit-stream comprising of an encoded source video information. The method may include generating a reconstructed video frame corresponding to the encoded source video frame by using the received bit-stream. The method may include obtaining one or more domain-transfer functions for each pixel block in the reconstructed video frame from the received bit-stream. The method may include obtaining one or more inverse functions by using the obtained one or more domain-transfer functions. The method may include generating one or more domain-transferred pixel blocks by applying the obtained one or more domain-transfer functions to each pixel block. The method may include generating one or more updated domain-transferred pixel blocks based on inputting the generated one or more domain-transferred pixel blocks to a pre-defined Artificial Intelligence (AI) model.

In an embodiment of the disclosure, the method may include generating an updated reconstructed video frame by applying the obtained one or more inverse functions on each updated domain-transferred pixel block.

In an embodiment of the disclosure, the one or more domain-transferred pixel blocks are generated by performing a domain transfer of one or more pixel blocks in the reconstructed video frame based on at least one of one or more histogram correction functions, and one or more spectral adjustment functions.

In an embodiment of the disclosure, the one or more histogram correction functions is obtained by processing one or more histograms of each pixel block of the reconstructed video frame, and a metadata comprising one or more histograms of a content used to train the pre-trained AI model, using a Cumulative Distribution Function (CDF) matching.

In an embodiment of the disclosure, the one or more spectral adjustment functions is obtained by determining an amount of scaling and shifting for matching one or more spectral components of the reconstructed pixel-block with the metadata comprising one or more spectral components of a content used to train the pre-trained AI model.

In an embodiment of the disclosure, the one or more histograms of each pixel block of the reconstructed video frame are computed separately for each color channel in the form of a one-dimensional Look-Up Table (1D-LUT) or for N channels in the form of a single N-dimensional LUT.

In an embodiment of the disclosure, the one or more spectral components of the reconstructed pixel-block are computed for one or more color channel in the form of transform-coefficients, using an invertible mathematical transformation, wherein the invertible mathematical transformation comprises at least one of a two-dimensional discrete cosine transform (DCT-II) or discrete wavelet transform (DWT).

In an embodiment of the disclosure, the one or more domain-transfer functions are signaled for each pixel block, frame, scene, or sequence level as determined by the video encoder using rate-distortion optimization (RDO).

In an embodiment of the disclosure, a method (400) for performing a content adaptive Artificial Intelligence (AI) based In-Loop Filtering (ILF) by a video encoder (310), is provided. The method may include generating a reconstructed video frame derived from a source video frame. The method may include computing one or more histograms representing pixel values of at least one of one or more color channels, and one or more frequency transform coefficients for each pixel block in the reconstructed video frame. The method may include obtaining a metadata comprising a histogram representing average pixel values of at least one of the one or more color channels, and one or more frequency transform coefficients of a content, where the content is used to train a pre-trained AI model of an ILF. The method may include comparing at least one of the computed one or more histograms, and the one or more frequency transform coefficients of each pixel block with the obtained metadata. The method may include determining one or more domain-transfer functions, and corresponding one or more inverse functions for each pixel block, based on a result of the comparing. The method may include generating one or more domain-transferred pixel blocks by applying the determined one or more domain-transfer functions to each pixel block. The method may include generating one or more updated domain-transferred pixel blocks based on inputting the generated one or more domain-transferred pixel blocks to the pre-defined AI model of the ILF. The method may include generating an updated reconstructed video frame by applying the corresponding one or more inverse functions on each updated domain-transferred pixel block.

In an embodiment of the disclosure, the one or more spectral components of the reconstructed pixel block are computed for one or more color channel in the form of the one or more transform coefficients using an invertible mathematical transformation, wherein the invertible mathematical transformation comprises at least one of a two-dimensional Discrete Cosine Transform (DCT-II), and a 2D Discrete Wavelet Transform (DWT).

In an embodiment of the disclosure, a method for transmitting a bit-stream generated for performing a content adaptive Artificial Intelligence (AI) based In-Loop Filtering (ILF) by a video encoder 310 is provided. The method may include generating a reconstructed video frame derived from a source video frame. The method may include computing one or more histograms representing pixel values of at least one of one or more color channels, and one or more frequency transform coefficients for each pixel block in the reconstructed video frame. The method may include obtaining a metadata comprising a histogram representing average pixel values of at least one of the one or more color channels, and one or more frequency transform coefficients of a content, where the content is used to train a pre-trained AI model of an ILF. The method may include comparing at least one of the computed one or more histograms, and the one or more frequency transform coefficients of each pixel block with the obtained metadata. The method may include determining one or more domain-transfer functions, and corresponding one or more inverse functions for each pixel block, based on a result of the comparing. The method may include generating one or more domain-transferred pixel blocks by applying the determined one or more domain-transfer functions to each pixel block. The method may include generating one or more updated domain-transferred pixel blocks based on inputting the generated one or more domain-transferred pixel blocks to the pre-defined AI model of the ILF. The method may include generating an updated reconstructed video frame by applying the corresponding one or more inverse functions on each updated domain-transferred pixel block.

The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and/or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of embodiments and examples, those skilled in the art will recognize that the embodiments and examples disclosed herein can be practiced with modification within the scope of the embodiments as described herein.

Claims

A method (600) for performing In-Loop Filtering (ILF) by a video decoder (320), comprising:

obtaining a bit-stream comprising of an encoded source video information;

generating a reconstructed video frame corresponding to the encoded source video frame by using the received bit-stream;

obtaining one or more domain-transfer functions for each pixel block in the reconstructed video frame from the received bit-stream;

obtaining one or more inverse functions by using the obtained one or more domain-transfer functions;

generating one or more domain-transferred pixel blocks by applying the obtained one or more domain-transfer functions to each pixel block;

generating one or more updated domain-transferred pixel blocks based on inputting the generated one or more domain-transferred pixel blocks to a pre-defined Artificial Intelligence (AI) model; and

generating an updated reconstructed video frame by applying the obtained one or more inverse functions on each updated domain-transferred pixel block.
The method (600) of claim 1, wherein the one or more domain-transferred pixel blocks are generated by performing a domain transfer of one or more pixel blocks in the reconstructed video frame based on at least one of one or more histogram correction functions, and one or more spectral adjustment functions.
The method (600) of claim 2, wherein the one or more histogram correction functions is obtained by processing one or more histograms of each pixel block of the reconstructed video frame, and a metadata comprising one or more histograms of a content used to train the pre-trained AI model, using a Cumulative Distribution Function (CDF) matching.
The method (600) of any one of claims 2 to 3, wherein the one or more spectral adjustment functions is obtained by determining an amount of scaling and shifting for matching one or more spectral components of the reconstructed pixel-block with the metadata comprising one or more spectral components of a content used to train the pre-trained AI model.
The method (600) of claim 3, wherein the one or more histograms of each pixel block of the reconstructed video frame are computed separately for each color channel in the form of a one-dimensional Look-Up Table (1D-LUT) or for N channels in the form of a single N-dimensional LUT.
The method (600) of claim 4, wherein the one or more spectral components of the reconstructed pixel-block are computed for one or more color channel in the form of transform-coefficients, using an invertible mathematical transformation, wherein the invertible mathematical transformation comprises at least one of a two-dimensional discrete cosine transform (DCT-II) or discrete wavelet transform (DWT).
The method (600) of any one of claims 1 to 6, wherein the one or more domain-transfer functions are signaled for each pixel block, frame, scene, or sequence level as determined by the video encoder using rate-distortion optimization (RDO).
A method (400) for performing a content adaptive Artificial Intelligence (AI) based In-Loop Filtering (ILF) by a video encoder (310), comprising:

generating a reconstructed video frame derived from a source video frame;

computing one or more histograms representing pixel values of at least one of one or more color channels, and one or more frequency transform coefficients for each pixel block in the reconstructed video frame;

obtaining a metadata comprising a histogram representing average pixel values of at least one of the one or more color channels, and one or more frequency transform coefficients of a content, where the content is used to train a pre-trained AI model of an ILF;

comparing at least one of the computed one or more histograms, and the one or more frequency transform coefficients of each pixel block with the obtained metadata;

determining one or more domain-transfer functions, and corresponding one or more inverse functions for each pixel block, based on a result of the comparing;

generating one or more domain-transferred pixel blocks by applying the determined one or more domain-transfer functions to each pixel block;

generating one or more updated domain-transferred pixel blocks based on inputting the generated one or more domain-transferred pixel blocks to the pre-defined AI model of the ILF; and

generating an updated reconstructed video frame by applying the corresponding one or more inverse functions on each updated domain-transferred pixel block.
The method (400) of claim 8, wherein the one or more domain-transferred pixel blocks are generated by performing a domain transfer of one or more pixel blocks in the reconstructed video frame based on at least one of one or more histogram correction functions, and one or more spectral adjustment functions.
The method (400) of claim 9, wherein the one or more histogram correction functions is obtained by processing one or more histograms of each pixel block of the reconstructed video frame, and a metadata comprising one or more histograms of a content used to train the pre-trained AI model, using a Cumulative Distribution Function (CDF) matching.
The method (600) of any one of claims 9 to 10, wherein the one or more spectral adjustment functions is obtained by determining an amount of scaling and shifting for matching one or more spectral components of the reconstructed pixel-block with the metadata comprising one or more spectral components of a content used to train the pre-trained AI model.
The method (400) of claim 10, wherein the one or more histograms of each pixel block of the reconstructed video frame are computed separately for each color channel in the form of a one-dimensional Look-Up Table (1D-LUT) or for N channels in the form of a single N-dimensional LUT.
The method (400) of claim 11, wherein the one or more spectral components of the reconstructed pixel block are computed for one or more color channel in the form of the one or more transform coefficients using an invertible mathematical transformation, wherein the invertible mathematical transformation comprises at least one of a two-dimensional Discrete Cosine Transform (DCT-II), and a 2D Discrete Wavelet Transform (DWT).
The method (400) of any one of claims 8 to 13, wherein the one or more domain-transfer functions are signaled for each pixel block, frame, scene, or sequence level as determined by the video encoder using rate-distortion optimization (RDO).
A method for transmitting a bit-stream generated for performing a content adaptive Artificial Intelligence (AI) based In-Loop Filtering (ILF) by a video encoder 310, comprising:

generating a reconstructed video frame derived from a source video frame;

computing one or more histograms representing pixel values of at least one of one or more color channels, and one or more frequency transform coefficients for each pixel block in the reconstructed video frame;

obtaining a metadata comprising a histogram representing average pixel values of at least one of the one or more color channels, and one or more frequency transform coefficients of a content, where the content is used to train a pre-trained AI model of an ILF;

comparing at least one of the computed one or more histograms, and the one or more frequency transform coefficients of each pixel block with the obtained metadata;

determining one or more domain-transfer functions, and corresponding one or more inverse functions for each pixel block, based on a result of the comparing;

generating one or more domain-transferred pixel blocks by applying the determined one or more domain-transfer functions to each pixel block;

generating one or more updated domain-transferred pixel blocks based on inputting the generated one or more domain-transferred pixel blocks to the pre-defined AI model of the ILF; and

generating an updated reconstructed video frame by applying the corresponding one or more inverse functions on each updated domain-transferred pixel block.