EP4646834A1

EP4646834A1 - Reducing complexity of video encoding and decoding

Info

Publication number: EP4646834A1
Application number: EP23915048.5A
Authority: EP
Inventors: Per Wennersten; Jacob STRÖM; Du LIU
Original assignee: Telefonaktiebolaget LM Ericsson AB
Current assignee: Telefonaktiebolaget LM Ericsson AB
Priority date: 2023-01-03
Filing date: 2023-12-04
Publication date: 2025-11-12
Also published as: WO2024147752A1; EP4646834A4; JP2025542460A; CN120391056A

Abstract

There is provided a method for encoding or decoding a video. The method comprises obtaining values of components of reconstructed samples. The method comprises obtaining first additional input data. The method comprises providing the obtained values of the components of the reconstructed samples to a first group of one or more convolution layers, thereby generating an output of the first group of convolution layers. The method further comprises providing the obtained first additional input data to a second group of one or more convolution layers, thereby generating an output of the second group of convolution layers. The method comprises encoding or decoding the video based on the output of the first group of convolution layers and the output of the second group of convolution layers. The first group of convolution layers and the second group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers.

Description

REDUCING COMPLEXITY OF VIDEO ENCODING AND DECODING

TECHNICAL FIELD

[0001] This disclosure relates to reducing complexity of video encoding and decoding.

BACKGROUND

[0002] Video is the dominant form of data traffic in today’s networks and is projected to continuously increase its share, as disclosed in P. Cerwall et al. Ericsson Mobility Report. https://www.ericsson.com/en/mobility-report, November 2019. One way to reduce the data traffic from video is compression. In the compression, the source video is encoded to a bitstream, which then can be stored and transmitted to end users. Using a decoder, the end user can extract the video data and display it on a screen.

[0003] However, since the encoder may not know what kind of device the encoded bitstream is going to be sent to, the encoder must compress the video into a standardized format. This way, all devices that support the chosen standard can successfully decode the video. Compression can be lossless, i.e., the decoded video will be identical to the source video that was given to the encoder, or lossy, where a certain degradation of content is accepted. Whether the compression is lossless or lossy has a significant impact on the bitrate, i.e., how high the compression ratio is, as factors such as noise can make lossless compression quite expensive.

[0004] A video sequence contains a sequence of pictures. A color space commonly used in video sequences is YCbCr, where Y is the luma (brightness) component, and Cb and Cr are the chroma components. Sometimes the Cb and Cr components are called U and V. Other color spaces are also used, such as ICtCp (a.k.a., IPT) (where I is the luma component, and Ct and Cp are the chroma components), constant-luminance YCbCr (where Y is the luma components, and Cb and Cr are the chroma components), RGB (where R, G, and B correspond to blue, green, and blue components respectively), YCoCg (where Y is the luma components, and Co and Cg are the chroma components), etc.

[0005] The order that the pictures are placed in the video sequence is called “display order.” Each picture is assigned with a Picture Order Count (POC) value to indicate its display order. In this disclosure, the terms “images,” “pictures” or “frames” are used interchangeably.

[0006] Video compression is used to compress video sequences into a sequence of coded pictures. In many existing video codecs, the picture is divided into blocks of different sizes. A block is a two-dimensional array of samples. The blocks serve as the basis for coding. A video decoder then decodes the coded pictures into pictures containing sample values.

[0007] Video standards are usually developed by international organizations as these represent different companies and research institutes with different areas of expertise and interests. The currently most applied video compression standard is H.264/AVC (Advanced Video Coding) which was jointly developed by ITU-T and ISO. The first version of H.264/AVC was finalized in 2003, with several updates in the following years. The successor of H.264/AVC, which was also developed by ITU-T (International Telecommunication Union - Telecommunication) and International Organization for Standardization (ISO), is known as H.265/HEVC (High Efficiency Video Coding) and was finalized in 2013. MPEG and ITU-T have created a successor to HEVC within the Joint Video Exploratory Team (JVET). The name of this video codec is Versatile Video Coding (VVC) and version 1 of the VVC specification has been published as Rec. ITU-T H.266 | ISO/IEC (International Electrotechnical Commission) 23090-3, “Versatile Video Coding”, 2020.

[0008] The VVC video coding standard is a block-based video codec and utilizes both temporal and spatial prediction. Spatial prediction is achieved using intra (I) prediction from within the current picture. Temporal prediction is achieved using uni-directional (P) or bidirectional inter (B) prediction at the block level from previously decoded reference pictures. In the encoder, the difference between the original pixel data and the predicted pixel data, referred to as the residual, is transformed into the frequency domain, quantized, and then entropy coded before being transmitted together with necessary prediction parameters such as prediction mode and motion vectors (which may also be entropy coded). The decoder performs entropy decoding, inverse quantization, and inverse transformation to obtain the residual, and then adds the residual to the intra or inter prediction to reconstruct a picture.

[0009] The VVC video coding standard uses a block structure referred to as quadtree plus binary tree plus ternary tree block structure (QTBT+TT), where each picture is first partitioned into square blocks called coding tree units (CTU). All CTUs are of the same size and the partitioning of the picture into CTUs is done without any syntax controlling it.

[0010] Each CTU is further partitioned into coding units (CUs) that can have either square or rectangular shapes. The CTU is first partitioned by a quad tree structure, then it may be further partitioned with equally sized partitions either vertically or horizontally in a binary structure to form coding units (CUs). A block could thus have either a square or rectangular shape. The depth of the quad tree and binary tree can be set by the encoder in the bitstream. The ternary tree (TT) part adds the possibility to divide a CU into three partitions instead of two equally sized partitions. This increases the possibilities to use a block structure that better fits the content structure of a picture, such as roughly following important edges in the picture.

[0011] A block that is intra coded is an I-block. A block that is uni-directional predicted is a P-block and a block that is bi-directional predicted a B-block. For some blocks, the encoder decides that encoding the residual is not necessary, perhaps because the prediction is sufficiently close to the original. The encoder then signals to the decoder that the transform coding of that block should be bypassed, i.e., skipped. Such a block is referred to as a skip-block.

[0012] At the 20^th JVET meeting, it was decided to setup an exploration experiment (EE) on neural network-based (NN-based) video coding. The exploration experiment continued at the 21^st and 22^nd JVET meetings with two EE tests: NN-based filtering and NN-based super resolution. In the 23^rd JVET meeting, the test was decided to be continued in three categories: enhancement filters, super-resolution methods, and intra prediction. In the category of enhancement filters, two configurations were considered: (i) the proposed filter used as in-loop filter and (ii) the proposed filter used as a post-processing filter.

[0013] VVC contains three in-loop filterings that are not based on neural networks: A deblocking filter, sample adaptive offsets (SAO) filter, and adaptive loop filter (ALF). The deblocking filter is used to remove block artifacts by smoothening discontinuities in horizontal and vertical directions across block boundaries. The deblocking filter uses a block boundary strength (BS) parameter to determine the filtering strength. The BS parameter can have values of 0, 1, and 2, where a larger value indicates a stronger filtering. The output of deblocking filter is further processed by SAO, and the output of the SAO is then processed by ALF operation. The output of the ALF can then be put into the display picture buffer (DPB), which is used for prediction of subsequently encoded (or decoded) pictures. Since the deblocking filter, the SAO filter, and the ALF influence the pictures in the DPB used for prediction, they are classified as in-loop filters, also known as loopfilters. It is possible for a decoder to further filter the image, but not send the filtered output to the DPB, but only to the display. In contrast to loopfilters, such a filter is not influencing future predictions and is therefore classified as a post-processing filter, also known as a postfilter.

[0014] The contributions JVET-X0066 described in EE1-1.6: Combined Test of EE1-1.2 and EE1-1.4, Y. Li, K. Zhang, L. Zhang, H. Wang, J. Chen, K. Reuze, A.M. Kotra, M. Karczewicz, JVET-X0066, Oct. 2021 and JVET-Y0143 described in EE1-1.2: Test on Deep InLoop Filter with Adaptive Parameter Selection and Residual Scaling, Y. Li, K. Zhang, L. Zhang, H. Wang, K. Reuze, A.M. Kotra, M. Karczewicz, JVET-Y0143, Jan. 2022 are two successive contributions that describe NN-based in-loop filtering. Both contributions use the same NN models for filtering. The NN-based in-loop filter is placed before SAO and ALF and the deblocking filter is turned off. The purpose of using the NN-based filter is to improve the quality of the reconstructed samples. Here it is helpful that the NN model is non-linear. While all of deblocking filter, SAO, and ALF contain non-linear elements such as conditions, and thus are not strictly linear, all three of them are based on linear filters. In contrast, a sufficiently big NN model in contrast can in principle learn any non-linear mapping and is therefore capable of representing a wider class of functions compared to deblocking, SAO and ALF.

[0015] In JVET-X0066 and JVET-Y0143, there are four NN models, i.e., four NN-based in-loop filters - one for luma intra samples, one for chroma intra samples, one for luma inter samples, and one for chroma inter samples. The use of NN filtering can be controlled on a block (CTU) level or a picture level. The encoder can determine whether to use NN filtering for each block or each picture.

[0016] The NN-based in-loop filters presented in JVET-X0066 and JVET-AB0053 increase the compression efficiency of the codec substantially, i.e., it lowers the bit rate substantially without lowering the objective quality as measured by MSE (mean-square error)- based PSNR (peak signal-to-noise ratio). Increases in compression efficiency, or simply “gain”, is often measured as the Bjontegaard-delta rate (BDR) against an anchor. As an example, a BDR of -1% means that the same PSNR can be reached with 1% fewer bits. As reported in JVET- Y0143, for the random access (RA) configuration, the BDR gain for the luma component (Y) is -9.80%, and for the all-intra (Al) configuration, the BDR gain for the luma component is -7.39%. The complexity of NN models used for compression is often measured by the number of Multiply- Accumulate (MAC) operations per pixel. The high gain of NN model is directly related to the high complexity of the NN model. The luma intra model described in JVET-Y0143 has a complexity of 430 kMAC/pixel, i.e., 430000 multiply-accumulate operations per pixel. There are also other measures of complexity, such as total model size in terms of stored parameters.

SUMMARY

[0017] Certain challenges presently exist. For example, since the high complexity of the NN filter is a major challenge for practical hardware implementations, reducing the complexity of the NN filter while preserving the performance of the NN filter (i.e., optimizing complexityperformance tradeoff) is highly desirable. However, the structure of the NN filter described above may not be optimal in terms of complexity-performance tradeoff.

[0018] In one example, the NN filter (the neural network loop filter) described above is configured to generate an improved output picture based on several inputs, but before these multiple inputs reach the “trunk” or “body” of the NN filter (where most of the computations of the NN filter occurs), these inputs are processed separately and then combined. These separate processing and combining corresponds to a substantial portion of the overall process of the NN filter. Therefore, there is a need to reduce this portion of the process of the NN filter.

[0019] Accordingly, according to a first aspect of the present disclosure, there is provided a method for encoding or decoding a video. The method comprises obtaining values of components of reconstructed samples; obtaining first additional input data; and providing the obtained values of the components of the reconstructed samples to a first group of one or more convolution layers, thereby generating an output of the first group of convolution layers. The method further comprises providing the obtained first additional input data to a second group of one or more convolution layers, thereby generating an output of the second group of convolution layers; and encoding or decoding the video based on the output of the first group of convolution layers and the output of the second group of convolution layers. The first group of convolution layers and the second group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers. [0020] According to a second aspect of the present disclosure, there is provided a computer program comprising instructions which when executed by processing circuitry cause the processing circuitry to perform the method according to the first aspect.

[0021] According to a third aspect of the present disclosure, there is provided a carrier containing the computer program of the above embodiment, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0022] In another aspect, there is provided an apparatus for encoding or decoding a video. The apparatus is configured to perform the method according to the first aspect.

[0023] Some embodiments of this disclosure provide a way to reduce the complexity of the NN model while substantially maintaining or improving the performance of the NN filter.

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0025] FIG. 1 A shows a system according to some embodiments.

[0026] FIG. IB shows a system according to some embodiments.

[0027] FIG. 1C shows a system according to some embodiments.

[0028] FIG. 2 shows a schematic block diagram of an encoder according to some embodiments.

[0029] FIG. 3 shows a schematic block diagram of a decoder according to some embodiments.

[0030] FIG. 4 shows a schematic block diagram of a portion of an NN filter according to some embodiments.

[0031] FIG. 5 shows a schematic block diagram of a portion of an NN filter according to some embodiments.

[0032] FIG. 6 shows computational operations performed by a convolution layer.

[0033] FIG. 7 shows a process according to some embodiments. [0034] FIG. 8 shows an apparatus according to some embodiments.

DETAILED DESCRIPTION

[0035] The following terminologies are used in the description of the embodiments below.

[0036] Neural network: a generic term for an entity with one or more layers of simple processing units called neurons or nodes having activation functions and interacting with each other via weighted connections and biases, which collectively create a tool in the context of nonlinear transforms.

[0037] Neural network architecture, network architecture, or architecture in short: the layout of a neural network describing the placement of the nodes and their connections, usually in the form of several interconnected layers, and may also specify the dimensionality of the input(s) and the output(s) as well as the activation functions for the nodes.

[0038] Neural network weights, or weights in short: The weight values assigned to the connections between the nodes in a neural network.

[0039] Neural network model, or model in short: a transform in the form of trained neural network(s). A neural network model may be specified with the neural network architecture, activation functions, biases, and/or weights.

[0040] Filter: A transforming entity. A neural network model is one realization of a filter. The term NN filter may be used as a short form of neural-network-based filter or neural network filter.

[0041] Neural network training, or training in short: The process of finding the values for the weights and biases for a neural network. Usually, a training data set is used to train the neural network and the goal of the training is to minimize a defined error. The amount of training data needs to be sufficiently large to avoid overtraining. Training a neural network is normally a timeconsuming task and typically comprises a number of iterations over the training data, where each iteration is referred to as an epoch.

[0042] FIG. 1A shows a system 100 according to some embodiments. The system 100 comprises a first entity 102, a second entity 104, and a network 110. The first entity 102 is configured to transmit towards the second entity 104 a video stream (a.k.a., “a video bitstream,” “a bitstream,” “an encoded video”) 106.

[0043] The first entity 102 may be any computing device (e.g., a network node such as a server) capable of encoding a video using an encoder 112 and transmitting the encoded video towards the second entity 104 via the network 110. The second entity 104 may be any computing device (e.g., a network node) capable of receiving the encoded video and decoding the encoded video using a decoder 114. Each of the first entity 102 and the second entity 104 may be a single physical entity or a combination of multiple physical entities. The multiple physical entities may be located in the same location or may be distributed in a cloud.

[0044] In some embodiments, as shown in FIG. IB, the first entity 102 is a video streaming server 132 and the second entity 104 is a user equipment (UE) 134. The UE 134 may be any of a desktop, a laptop, a tablet, a mobile phone, or any other computing device. The video streaming server 132 is capable of transmitting a video bitstream 136 (e.g., YouTube™ video streaming) towards the video streaming client 134. Upon receiving the video bitstream 136, the UE 134 may decode the received video bitstream 136, thereby generating and displaying a video for the video streaming.

[0045] In other embodiments, as shown in FIG. 1C, the first entity 102 and the second entity 104 are first and second UEs 152 and 154. For example, the first UE 152 may be an offeror of a video conferencing session or a caller of a video chat, and the second UE 154 may be an answerer of the video conference session or the answerer of the video chat. In the embodiments shown in FIG. 1C, the first UE 152 is capable of transmitting a video bitstream 156 for a video conference (e.g., Zoom™, Skype™, MS Teams™, etc.) or a video chat (e.g., Facetime™) towards the second UE 154. Upon receiving the video bitstream 156, the UE 154 may decode the received video bitstream 156, thereby generating and displaying a video for the video conferencing session or the video chat.

[0046] FIG. 2 shows a schematic block diagram of the encoder 112 according to some embodiments. The encoder 112 is configured to encode a block of sample values (hereafter “block”) in a video frame of a source video 202. In the encoder 112, a current block (e.g., a block included in a video frame of the source video 202) is predicted by performing a motion estimation by a motion estimator 250 from an already provided block in the same frame or in a previous frame. The result of the motion estimation is a motion or displacement vector associated with the reference block, in the case of inter prediction. The motion vector is utilized by the motion compensator 250 for outputting an inter prediction of the block.

[0047] An intra predictor 249 computes an intra prediction of the current block. The outputs from the motion estimator/compensator 250 and the intra predictor 249 are inputted to a selector 251 that either selects intra prediction or inter prediction for the current block. The output from the selector 251 is input to an error calculator in the form of an adder 241 that also receives the sample values of the current block. The adder 241 calculates and outputs a residual error as the difference in sample values between the block and its prediction. The error is transformed in a transformer 242, such as by a discrete cosine transform, and quantized by a quantizer 243 followed by coding in an encoder 244, such as by entropy encoder. In inter coding, the estimated motion vector is brought to the encoder 244 for generating the coded representation of the current block.

[0048] The transformed and quantized residual error for the current block is also provided to an inverse quantizer 245 and inverse transformer 246 to retrieve the original residual error. This error is added by an adder 247 to the block prediction output from the motion compensator 250 or the intra predictor 249 to create a reconstructed sample block 280 that can be used in the prediction and coding of a next block. The reconstructed sample block 280 is processed by a NN filter 230 (a.k.a., “neural network loop filter” or “NNLF”) according to the embodiments in order to perform filtering to combat any blocking artifact. The output from the NN filter 230, i.e., the output data 290, is then temporarily stored in a frame buffer 248, where it is available to the intra predictor 249 and the motion estimator/compensator 250.

[0049] In some embodiments, the encoder 112 may include SAO unit 270 and/or ALF 272. The SAO unit 270 and the ALF 272 may be configured to receive the output data 290 from the NN filter 230, perform additional filtering on the output data 290, and provide the filtered output data to the buffer 248.

[0050] Even though, in the embodiments shown in FIG. 2, the NN filter 230 is disposed between the SAO unit 270 and the adder 247, in other embodiments, the NN filter 230 may replace the SAO unit 270 and/or the ALF 272. Alternatively, in other embodiments, the NN filter 230 may be disposed between the buffer 248 and the motion compensator 250. Furthermore, in some embodiments, a deblocking filter (not shown) may be disposed between the NN filter 230 and the adder 247 such that the reconstructed sample block 280 goes through the deblocking process and then is provided to the NN filter 230.

[0051] FIG. 3 is a schematic block diagram of the decoder 114 according to some embodiments. The decoder 114 comprises a decoder 361, such as entropy decoder, for decoding an encoded representation of a block to get a set of quantized and transformed residual errors. These residual errors are dequantized in an inverse quantizer 362 and inverse transformed by an inverse transformer 363 to get a set of residual errors. These residual errors are added in an adder 364 to the sample values of a reference block. The reference block is determined by a motion estimator/compensator 367 or intra predictor 366, depending on whether inter or intra prediction is performed.

[0052] A selector 368 is thereby interconnected to the adder 364 and the motion estimator/compensator 367 and the intra predictor 366. The resulting decoded block 380 output form the adder 364 is input to a NN filter unit 330 according to the embodiments in order to filter any blocking artifacts. The filtered block 390 is output form the NN filter 330 and is furthermore preferably temporarily provided to a frame buffer 365 and can be used as a reference block for a subsequent block to be decoded.

[0053] The frame buffer (e.g., decoded picture buffer (DPB)) 365 is thereby connected to the motion estimator/compensator 367 to make the stored blocks of samples available to the motion estimator/compensator 367. The output from the adder 364 is preferably also input to the intra predictor 366 to be used as an unfiltered reference block.

[0054] In some embodiments, the decoder 114 may include SAO unit 380 and/or ALF 372. The SAO unit 380 and the ALF 382 may be configured to receive the output data 390 from the NN filter 330, perform additional filtering on the output data 390, and provide the filtered output data to the buffer 365.

[0055] Even though, in the embodiments shown in FIG. 3, the NN filter 330 is disposed between the SAO unit 380 and the adder 364, in other embodiments, the NN filter 330 may replace the SAO unit 380 and/or the ALF 382. Alternatively, in other embodiments, the NN filter 330 may be disposed between the buffer 365 and the motion compensator 367. Furthermore, in some embodiments, a deblocking filter (not shown) may be disposed between the NN filter 330 and the adder 364 such that the reconstructed sample block 380 goes through the deblocking process and then is provided to the NN filter 330.

[0056] FIG. 4 is a schematic block diagram of a portion the NN filter 230/330 for filtering intra luma samples according to some embodiments. In this disclosure, luma (or chroma) intra samples are luma (or chroma) components of samples that are intra-predicted. Similarly, luma (or chroma) inter samples are luma (or chroma) components of samples that are inter-predicted.

[0057] As shown in FIG. 4, the NN filter 230/330 may have four inputs: (1) values of luma components of reconstructed samples (“rec”) 280/380; (2) values of luma components of predicted samples (“pred”) 295/395; (3) block boundary strength, BBS, information indicating strength of a filtering applied to a boundary of luma components of samples (“bs”); and (4) quantization parameters (“qp”). In some embodiments, additional input(s) (e.g., partition information indicating how luma components of samples are partitioned) may be additionally used as the input for the NN filter 230/330.

[0058] Each of the four inputs may go through a convolution layer (labelled as “conv3x3” in FIG. 4) and a parametric rectified linear unit (PReLU) layer (labelled as “PReLU”) separately. The four outputs from the four PReLU layers may then be concatenated via a concatenating unit (labelled as “concat” in FIG. 4) and fused together to generate data (a.k.a., “signal”) “y ” The convolution layer “conv3x3” is a convolutional layer with kernel size 3x3 and the convolution layer “convlxl” is a convolutional layer with kernel size 1x1. The PReLUs may make up the activation layer.

[0059] In some embodiments, qp may be a scalar value. In such embodiments, the NN filter 230/330 may also include a dimension manipulation unit (labelled as “Unsqueeze expand” in FIG. 4) that may be configured to expand qp such that the expanded qp has the same size as other inputs (i.e., rec, pred, and bs). However, in other embodiments, qp may be a matrix of which the size may be same as the size of other inputs (e.g., rec, pred, and bs). For example, different samples inside a CTU may be associated with a different qp value. In such embodiments, the dimension manipulation unit is not needed.

[0060] In some embodiments, the NN filter 230/330 may also include a downsampler (labelled as “2J,” in FIG. 4) which is configured to perform a downsampling with a factor of 2. [0061] As shown in FIG. 4, the data “y” may be provided to a group of N sequential attention residual (herein after, “AR”) blocks 402. In some embodiments, the N sequential AR blocks 402 may have the same structure while, in other embodiments, they may have different structures. N may be any integer that is greater than or equal to 2. For example, N may be equal to 8.

[0062] As shown in FIG. 4, the first AR block 402 included in the group may be configured to receive the data “y” and generate first output data “zo.” The second AR block 402 which is disposed right after the first AR block 402 may be configured to receive the first output data “zo” and generate second output data “zi.”

[0063] In case the group includes only two AR blocks 402 (i.e., the aforementioned first and second AR blocks), the second output data “zi” may be provided to a final processing unit 550 (shown in FIG. 5) of the NN filter 230/330.

[0064] On the other hand, in case the group includes more than two AR blocks, each AR block 402 included in the group except for the first and the last AR blocks may be configured to receive the output data from the previous AR block 402 and provide its output data to the next AR block. The last AR block 402 may be configured to receive the output data from the previous AR block and provide its output data to the final processing unit 550 of the NN filter 230/330. In FIG. 4, N-l corresponds to the number of AR blocks included in the NN filter 230/330.

[0065] In some embodiments, some or all of the AR blocks 402 may include a spatial attention block 412 which is configured to generate attention mask f. The attention mask f may have one channel and its size may be the same as the data “y.” Taking the first AR block 402 as an example, the spatial attention block included in the first AR block 402 may be configured to multiply the attention mask f with the residual data “r” to obtain data “r .” The data “rf ’ may be combined with the residual data “r” and then combined with the data “y”, thereby generating first output data “zo.”

[0066] As shown in FIG. 5, the output ZN-I of the group of the AR blocks 402 may be processed by a convolution layer 502, a PReLU 504, another convolution layer 506, pixel shuffling (or really sample shuffling) 508, and a final scaling 510, thereby generating the filtered output data (“Output”) 290/390. [0067] Referring back to FIG. 4, the NN filter 230/330 may include four separate first layers 452-458. As shown in FIG. 4, each of the four first layers 452-458 may include a 3 X 3 convolution layer and a PReLU. Each of the four first layers 452-458 may be configured to receive different input data. For example, the layer 452, 454, 456, and 458 may be configured to receive the input data “rec,” “pred,” “bs,” and “qp,” respectively.

[0068] The 3 x 3 convolution layer included in each of the first layers 452-458 may include 96 channels (i.e., 96 3 X 3 kernel filters), and thus, for a single input (that comprises a plurality of pixel values) provided to each of the first layers 452-458, there would be 96 outputs (i.e., one output corresponding to each of the 96 channels). Therefore, the first layers 452-458 may be configured to generate 4 x 96 outputs (a.k.a., output channels) for the four inputs (i.e., “rec,” “pred,” “bs,” and “qp”). These 4 x 96 outputs may be processed via the concatenate unit, the fuse unit, and the transition unit, and thus combined into j' corresponding to 96 outputs.

[0069] In general, a rough way to calculate the number of multiply-accumulate (MAC) operations needed for a convolutional neural network layer is to multiply the number of inputs (a.k.a., input channels), the number of filter coefficients, and the number of outputs (a.k.a., output channels).

[0070] For example, as shown in FIG. 6, let’s assume that the input “rec” is an image comprising 9 pixel values. Note that this number is provided just for simple explanation purpose, and does not limit the embodiments of this disclosure in any way. For each pixel value included in the input “rec,” 96 different kernel filters 602 is applied. For example, if a kernel filter 602 of the convolution layer included in the first layer 452 is applied to the pixel value a22 of the input “rec,” a filtered value corresponding to the pixel value a22 can be calculated as follows: an x wn + ai2 X W12 + ai3 X W13 + a21 X W21 + a22 X W22 + a23 X W23 + a31 X W31 + a32 X W32 + a33 X W33.

[0071] Thus, for each pixel value, 9 multiplication operations (corresponding to the number of filter coefficients wn, W12, ...) are performed 96 times (# of kernel filters which corresponds to the # of output channels). Therefore, here, the number of MAC operations performed for each pixel for the first layer 452 (which roughly indicates the computational complexity of the first layer 452) is roughly 1 X (3 X 3) X 96 = 864 MAC operations. In case, all of the layers 452-458 have the same # of filtering channels (i.e., # of kernel filters), the total number of MAC operations needed for the four layers 452-458 would be 864 X 4 = 3456.

[0072] As shown in FIG. 4, the outputs of the first layers 452-458 are provided to a second layer 462 of the NN filter 230/330. The first operation (the operation performed by the fusing unit) of the second layer 462 is a 1 x 1 convolution that is followed by a second operation (the operation performed by the transitioning unit) of the second layer 462 which is a 3 X 3 convolution of reduced scale.

[0073] For the first operation of the second layer 462, the number of MAC operations may be calculated as follows: For the first operation of the second layer 462, the number of inputs is 96 x 4 = 384 since all outputs of the first layers 452-458 are provided to the second layer 462. Also, for the first operation of the second layer 462, the number of filter coefficients is 1 (since 1 X 1 kernel filter is used in the fusing unit), and the number of outputs from the 1 X 1 kernel filter is 96. Thus, the number of MAC operations performed for the first operation of the second layer 462 is 384 X 1 X 1 X 96 = 36864.

[0074] Similarly, the number of MAC operations for the second operation of the second layer 462 may be calculated as follows: For the second operation of the second layer 462, the number of inputs (i.e., the number of outputs from the first operation of the second layer 462) is 96, the number of filter coefficients is 9 (since 3 x 3 kernel filter is used in the 3 x 3 convolution layer), and the number of outputs is 96. Thus, the number of MAC operations performed for the second operation of the second layer 462 is 96 x 3 x 3 x 96 x x Q = 20736 MAC operations, where the multiplication by - twice is due to the fact that the output is downsampled in the x- and y-directions, meaning that only every fourth output sample needs to be calculated.

[0075] In a summary, in case 96 filtering channels is used for each of the four inputs, the total number of MAC operations performed for generating the output y in FIG. 4 is 3456 + 36864 + 20736 = 61056 MAC operations.

[0076] In the embodiments described above, all inputs (e.g., “rec,” “pred,” “bs,” and “qp”) are given the same level of importance. But the different inputs may not be equally important. For example, the input “rec” corresponds to pixel values of a reconstructed image. Since this input is the input that needs to be improved (meaning that this input is the focus of the filtering process performed by the NN filter 230/330), it makes sense to put more emphasis on this input as compared to other inputs during the filtering process.

[0077] Thus, according to some embodiments, different levels of importance may be assigned to different inputs such that different numbers of MAC operations are performed for different inputs. More specifically, in some embodiments, more importance may be given to the inputs that are more important to the end result of the filtering process by configuring a higher number of MAC operations for those important input(s) and a lower number of MAC operations for those less important input(s).

[0078] In some embodiments, different inputs are assigned to different levels of importance by using different numbers of filtering channels for the different inputs. For example, the number of filtering channels applied to the input “rec” (the most important input from among the four inputs) can be increased from 96 to 192, thereby putting a greater emphasis on the input “rec.” On other hand, the number of filtering channels applied to the input “pred” may be reduced from 96 to 48, and the number of filtering channels applied to each of the input “bs” and the input “qp” may be reduced from 96 to 24.

[0079] Changing the numbers of filtering channels applied to different inputs may result in reducing the number of MAC operations performed for the NN filtering. More specifically, in the above example, the number of MAC operations for the input “rec” would be 1 x (3 x 3) x 192 = 1728, the number of MAC operations for the input “pred” would be 1 X (3 X 3) X 48 = 432, and the number of MAC operations for each of the input “bs” and “qp” would be 1 x (3 x 3) x 24 = 216. Thus, in this example, the total number of the MAC operations performed by the head of the NN filter 230/330 (i.e., the top portion of the NN filter 230/330 shown in FIG. 4) is 1728 + 432 + 216 + 216 = 2592. The table below shows a difference in the total number of MAC operations performed in case the same number of filtering channels is used for the different inputs and the different numbers of filtering channels is used for the different inputs.

[0080] The change of the numbers of filtering channels applied to the different inputs may result in a bigger change on the next stage — the second layer 462 of the NN filter 230/330. For example, in the above example, the number of the inputs (i.e., the number of the input channels) for the second layer 462 of the NN filter 230/330 is 192 + 48 + 24 + 24 = 288 (as compared to (96 x 4) = 384). Thus, the number of MAC operations performed by the 1 X 1 convolution layer included in the fusing unit of the second layer 462 is 288 X 1 X 1 X 96 = 27648 MAC operations (as compared to 384 x 1 x 1 x 96). Also, in the above example, the number of inputs (i.e., the number of input channels) for the 3 X 3 convolution layer included in the transitioning unit of the second layer 462 of the NN filter 230/330 is 96, and thus the number of MAC operations performed by the 3 x 3 convolution layer is 96 X 3 X 3 X 96 = 20736 MAC operations.

[0081] Thus, the total number of MAC operations performed by the head part of the NN filter 230/330 has decreased from 61056 MAC operations to 50976 MAC operations — a reduction of 16.5%. The entire network (including the body and tail) involves around 500k MAC operations, so the change of the head is equivalent to a reduction of about (61056-50976)/500000 = 2% reduction in overall kMACs/pixel for the network. At the same time, the increased focus on the important reconstruction input leads to slightly increasing compression efficiency by -0.03%.

[0082] In some embodiments, instead of changing the number of filtering channels applied to each of the inputs “rec,” “pred,” “bs,” and “qp” to 192, 24, 12, and 12, the number of filtering channels may be changed to 192, 24, 12, and 12 for the inputs “rec,” “pred,” “bs,” and “qp,” respectively. Based on the above described method of calculating MAC operations, changing the numbers of filtering channels used in the head part of the NN filter 230/330 gives a reduction of 4% in overall kMACs/pixel for the entire model.

[0083] In some embodiments, in addition to changing the numbers of filtering channels as discussed above, the number of channels between the fusing and transitioning layers may be reduced from 96 to 48. This leads to a total reduction of 9% kMACs/pixel for the model, and a BD-rate penalty of only +0.03%.

[0084] Note that, in the embodiments described above, the NN filter 230/330 is an in-loop filter, meaning its output can be used to predict future pictures in the video. However, a similar or identical neural network can be used as a post filter instead, for filtering pictures after they have been used for prediction.

[0085] In some embodiments, in addition to or instead of changing the numbers of filtering channels, a different set of inputs may be provided to the NN filter 230/330. More specifically, in some embodiments, instead of the four inputs (“rec,” “pred,” “bs,” and “qp”), additional inputs such as block size, motion vectors, and/or prediction mode may be provided. In other embodiments, instead of some of the inputs “pred,” “bs,”, and “qp,” any one or more additional inputs may be provided. In the above discussed embodiments, each of these additional or alternative inputs may also have a reduced number of channels compared to the reconstructed picture input. For example, the first layers of the NN filter 230/330 may receive five inputs - the four inputs shown in FIG. 4 and the fifth input identifying a block type (I/P/B - intra coded/uni- predicted/bi-predicted) where 192 filtering channels are used for the input “rec,” 24 filtering channels are used for the input “pred,” and 12 filtering channels are used for the remaining three inputs.

[0086] In some embodiments, in addition to changing the numbers of filtering channels applied to the different inputs, the size of the kernel filter used for filtering may also be changed. For example, instead of using a 3 X 3 kernel filter for the input “rec,” a first size of a kernel filter (e.g., 5 x 5) may be used for filtering the input “rec” while a second size of a kernel filter (3 x 3) is used for filtering other inputs where the first size is larger than the second size. [0087] In some embodiments, the first layer 452 for filtering the input “rec” may use kernel filters having different sizes to filter the input “rec.” For example, in case there are 196 filtering channels in the first layer 452, the first 96 filtering channels in the 196 filtering channels may use kernel filters each having a first size (e.g., 5 X 5), but the remaining filtering channels in the 196 filtering channels may use kernel filters each having a second size (3 x 3) where the first size and the second size are different. In some embodiments, the first size is larger than the second size.

[0088] In some embodiments, more emphasis may be placed on some input (e.g., input “rec”) by including more than one convolution layer in the first layer (e.g., 452). For example, while, in FIG. 4, the first layer 452 includes a single 3 x 3 convolution layer followed by a single PreLU, in some embodiments, the first layer 452 includes multiple 3 x 3 convolution layers and multiple PreLUs. More specifically, in such example, the first layer 452 may include a first 3 x 3 convolution layer followed by a first PreLU followed by a second 3 x 3 convolution layer followed by a second PreLU.

[0089] In some embodiments, more emphasis may be placed on some input (e.g., input “rec”) by increasing the number of bits for the kernel filter(s). For example, a first number of bits (e.g., 14) may be used to indicate filter weights (e.g., Wn, W12, . . . shown in FIG. 6) of the kernel filter of the 3 X 3 convolution layer of the first layer 452 while a second number of bits (e.g., 7) may be used to indicate filter weights of the kernel filter of 3 x 3 convolution layer of the first layers 454, 456, and/or 458 where the first number is greater than the second number.

[0090] In some embodiments, the input “rec” corresponds to pixel values of a reconstructed luma picture or pixel values of a reconstructed chroma picture. However, in other embodiments, the input “rec” corresponds to both the pixel values of the reconstructed luma picture and the pixel values of the reconstructed chroma picture. In such embodiments, different emphasis may be placed on the pixel values of the reconstructed luma picture and the pixel values of the reconstructed chroma picture. For example, if the NN filter 230/330 is used for predicting chroma samples, more emphasis may be placed on the pixel values of the reconstructed chroma samples. The aforementioned techniques (e.g., increasing a number of filtering channels) may be used for placing more emphasis on pixel values of the reconstructed chroma samples.

[0091] In a summary, some embodiments described above are related to increasing, for a particular input (e.g., input “rec”), the number of filtering channels, the size of a kernel filter, and/or the number of bits used for indicating filter weights of a kernel filter and/or using additional convolution layers while decreasing or maintaining, for other inputs, the number of filtering channels, the size of a kernel filter, the number of bits used for indicating filter weights of a kernel filter, and/or the number of convolution layers. Through these selective adjustments, more emphasis can be placed on more important input, and thus the accuracy of the filtering operation of the NN filter 230/330 can be increased or maintained while reducing the complexity of the filtering operation.

[0092] FIG. 7 shows a process 700 for encoding or decoding a video, according to some embodiments. FIG. 7 begins with step s702. Step s702 comprises obtaining values of components of reconstructed samples. Step s704 comprises obtaining first additional input data. Step s706 comprises providing the obtained values of the components of the reconstructed samples to a first group of one or more convolution layers, thereby generating an output of the first group of convolution layers. Step s708 comprises providing the obtained first additional input data to a second group of one or more convolution layers, thereby generating an output of the second group of convolution layers. Step s710 comprises encoding or decoding the video based on the output of the first group of convolution layers and the output of the second group of convolution layers. The first group of convolution layers and the second group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers.

[0093] In some embodiments, process 700 comprises combining the output of the first group of convolution layers and the output of the second group of convolution layers, thereby generating a combined output; providing the combined output to a third group of one or more convolution layers, thereby generating an output of the third group of convolution layers, wherein the video is encoded or decoded based on the output of the third group of convolution layers.

[0094] In some embodiments, the first additional input data is any one of values of components of predicted samples, partitional information indicating how components of samples are partitioned, block boundary strength information indicating strength of a filtering applied to a boundary of components of samples, values of quantization parameters, block size information indicating a block size, motion vector information indicating motion vectors, prediction mode information indicating a prediction mode, and/or block type information indicating a block type. [0095] In some embodiments, a number of channels included in the first group of convolution layers is more than twice of a number of channels included in the second group of convolution layers.

[0096] In some embodiments, the number of channels included in the first group of convolution layers is 192, and the number of channels included in the second group of convolution layers is 48, 24, or 12.

[0097] In some embodiments, process 700 comprises obtaining second additional input data; and providing the second additional input data to a fourth group of one or more convolution layers, thereby generating an output of the fourth group of convolution layers, wherein the video is encoded or decoded based on the output of the fourth group of convolution layers, the first group of convolution layers and the fourth group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers, and the second group of convolution layers and the fourth group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers.

[0098] In some embodiments, the number of channels included in the second group of convolution layers is 48 or 24, and a number of channels included in the fourth group of convolution layers is 24 or 12.

[0099] In some embodiments, a size of a kernel filter for the first group of convolution layers is different from a size of a kernel filter for the second group of convolution layers.

[0100] In some embodiments, the size of the kernel filter for first group of convolution layers is 5 x 5, and the size of the kernel filter for the second group of convolution layers is 3 x 3.

[0101] In some embodiments, a convolution layer of the first group of convolution layers comprises a set of channels, a first subset of channels included in the set of channels has a kernel filter having a first size, a second subset of channels included in the set of channels has a kernel filter having a second size, and the first size and the second size are different.

[0102] In some embodiments, the size of the kernel filter of the first subset of channels is 5 x 5, and the size of the kernel filter of the second subset of channels is 3 x 3.

[0103] In some embodiments, the first group of convolution layers comprises a first number of convolution layers, the second group of convolution layers comprises a second number of convolution layers, and the first number is greater than the second number.

[0104] In some embodiments, the first number is 2 and the second number is 1.

[0105] In some embodiments, the first group of convolution layers includes a first convolution layer and a second convolution layer, the first convolution layer has a kernel filter having a first size, the second convolution layer has a kernel filter having a second size, and the first size and the second size are different.

[0106] In some embodiments, the first size is 5 x 5, and the second size is 3 x 3.

[0107] In some embodiments, the first group of convolution layers has a kernel filter having a first plurality of filter values, the second group of convolution layers has a kernel filter having a second plurality of filter values, and a number of bits of each filter value included in the first plurality of filter values is greater than a number of bits of each filter value included in the second plurality of filter values.

[0108] In some embodiments, the number of bits of each filter value included in the first plurality of filter values is 14 bits, and the number of bits of each filter value included in the second plurality of filter values is 7 bits.

[0109] In some embodiments, the reconstructed samples are one of reconstructed luma samples and reconstructed chroma samples. Process 700 may comprise obtaining values of components of another one of the reconstructed luma samples and the reconstructed chroma samples; providing to a fifth group of one or more convolution layers the obtained values of the components of said another one of the reconstructed luma samples and the reconstructed chroma samples; and encoding or decoding the video based on the output of the fifth group of convolution layers, wherein the first group of convolution layers and the fifth group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers.

[0110] FIG. 8 is a block diagram of an apparatus 800 for implementing the encoder 112, the decoder 114, or a component included in the encoder 112 or the decoder 114 (e.g., the NN filter 280 or 330), according to some embodiments. When apparatus 800 implements a decoder, apparatus 800 may be referred to as a “decoding apparatus 800,” and when apparatus 800 implements an encoder, apparatus 800 may be referred to as an “encoding apparatus 800.” As shown in FIG. 8, apparatus 800 may comprise: processing circuitry (PC) 802, which may include one or more processors (P) 855 (e.g., a general purpose microprocessor and/or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., apparatus 800 may be a distributed computing apparatus); at least one network interface 848 comprising a transmitter (Tx) 845 and a receiver (Rx) 847 for enabling apparatus 800 to transmit data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network) to which network interface 848 is connected (directly or indirectly) (e.g., network interface 848 may be wirelessly connected to the network 110, in which case network interface 848 is connected to an antenna arrangement); and a storage unit (a.k.a., “data storage system”) 808, which may include one or more non-volatile storage devices and/or one or more volatile storage devices. In embodiments where PC 802 includes a programmable processor, a computer program product (CPP) 841 may be provided. CPP 841 includes a computer readable medium (CRM) 842 storing a computer program (CP) 843 comprising computer readable instructions (CRI) 844. CRM 842 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 844 of computer program 843 is configured such that when executed by PC 802, the CRI causes apparatus 800 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, apparatus 800 may be configured to perform steps described herein without the need for code. That is, for example, PC 802 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and/or software.

[0111] Summary of Embodiments

Al. A method (700) for encoding or decoding a video, the method comprising: obtaining (s702) values of components of reconstructed samples; obtaining (s704) first additional input data; providing (s706) the obtained values of the components of the reconstructed samples to a first group of one or more convolution layers, thereby generating an output of the first group of convolution layers; providing (s708) the obtained first additional input data to a second group of one or more convolution layers, thereby generating an output of the second group of convolution layers; and encoding or decoding (s710) the video based on the output of the first group of convolution layers and the output of the second group of convolution layers, wherein the first group of convolution layers and the second group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers.

Al-1. The method of embodiment Al, the method comprising: combining the output of the first group of convolution layers and the output of the second group of convolution layers, thereby generating a combined output; providing the combined output to a third group of one or more convolution layers, thereby generating an output of the third group of convolution layers, wherein the video is encoded or decoded based on the output of the third group of convolution layers.

A2. The method of embodiment Al or Al-1, wherein the first additional input data is any one of values of components of predicted samples, partitional information indicating how components of samples are partitioned, block boundary strength information indicating strength of a filtering applied to a boundary of components of samples, values of quantization parameters, block size information indicating a block size, motion vector information indicating motion vectors, prediction mode information indicating a prediction mode, and/or block type information indicating a block type.

A3. The method of any one of embodiments Al-Al-1, wherein a number of channels included in the first group of convolution layers is more than twice of a number of channels included in the second group of convolution layers. A4. The method of embodiment A3, wherein the number of channels included in the first group of convolution layers is 192, and the number of channels included in the second group of convolution layers is 48, 24, or 12.

A5. The method of any one of embodiments A1-A4, the method comprising: obtaining second additional input data; and providing the second additional input data to a third group of one or more convolution layers, thereby generating an output of the third group of convolution layers, wherein the video is encoded or decoded based on the output of the third group of convolution layers, the first group of convolution layers and the third group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers, and the second group of convolution layers and the third group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers.

A6. The method of embodiment A5 (when embodiment A5 depends on embodiment A4), wherein the number of channels included in the second group of convolution layers is 48 or 24, and a number of channels included in the third group of convolution layers is 24 or 12.

A7. The method of any one of embodiments A1-A6, wherein a size of a kernel filter for the first group of convolution layers is different from a size of a kernel filter for the second group of convolution layers.

A8. The method of embodiment A7, wherein the size of the kernel filter for first group of convolution layers is 5 x 5, and the size of the kernel filter for the second group of convolution layers is 3 x 3. A9. The method of any one of embodiments A1-A8, wherein a convolution layer of the first group of convolution layers comprises a set of channels, a first subset of channels included in the set of channels has a kernel filter having a first size, a second subset of channels included in the set of channels has a kernel filter having a second size, and the first size and the second size are different.

A10. The method of embodiment A9, wherein the size of the kernel filter of the first subset of channels is 5 x 5, and the size of the kernel filter of the second subset of channels is 3 x 3.

Al l. The method of any one of embodiments A1-A10, wherein the first group of convolution layers comprises a first number of convolution layers, the second group of convolution layers comprises a second number of convolution layers, and the first number is greater than the second number.

A12. The method of embodiment Al l, the first number is 2 and the second number is 1.

A13. The method of embodiment Al 1 or A12, wherein the first group of convolution layers includes a first convolution layer and a second convolution layer, the first convolution layer has a kernel filter having a first size, the second convolution layer has a kernel filter having a second size, and the first size and the second size are different.

A14. The method of embodiment A13, wherein the first size is 5 x 5, and the second size is 3 x 3. A15. The method of any one of embodiments A1-A14, wherein the first group of convolution layers has a kernel filter having a first plurality of filter values, the second group of convolution layers has a kernel filter having a second plurality of filter values, and a number of bits of each filter value included in the first plurality of filter values is greater than a number of bits of each filter value included in the second plurality of filter values.

A16. The method of embodiment A15, wherein the number of bits of each filter value included in the first plurality of filter values is 14 bits, and the number of bits of each filter value included in the second plurality of filter values is 7 bits.

Al 7. The method of any one of embodiments Al -Al 6, wherein the reconstructed samples are one of reconstructed luma samples and reconstructed chroma samples, the method comprises: obtaining values of components of another one of the reconstructed luma samples and the reconstructed chroma samples; providing to a third group of one or more convolution layers the obtained values of the components of said another one of the reconstructed luma samples and the reconstructed chroma samples; and encoding or decoding the video based on the output of the third group of convolution layers, and the first group of convolution layers and the third group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers. Bl. A computer program (800) comprising instructions (844) which when executed by processing circuitry (802) cause the processing circuitry to perform the method of any one of embodiments Al -Al 7.

B2. A carrier containing the computer program of embodiment Bl, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

Cl. An apparatus (800) for encoding or decoding a video, the apparatus being configured to: obtain (s702) values of components of reconstructed samples; obtain (s704) first additional input data; provide (s706) the obtained values of the components of the reconstructed samples to a first group of one or more convolution layers, thereby generating an output of the first group of convolution layers; provide (s708) the obtained first additional input data to a second group of one or more convolution layers, thereby generating an output of the second group of convolution layers; and encode or decode (s710) the video based on the output of the first group of convolution layers and the output of the second group of convolution layers, wherein the first group of convolution layers and the second group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers.

C2. The apparatus of embodiment Cl, wherein the apparatus is further configured to perform the method of any one of embodiments A2-A17.

DI. An apparatus (800) comprising: a processing circuitry (802); and a memory (841), said memory containing instructions executable by said processing circuitry, whereby the apparatus is operative to perform the method of any one of embodiments A1-A17. [0112] Conclusion

[0113] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0114] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

Claims

1. A method (700) for encoding or decoding a video, the method comprising: obtaining (s702) values of components of reconstructed samples; obtaining (s704) first additional input data; providing (s706) the obtained values of the components of the reconstructed samples to a first group of one or more convolution layers, thereby generating an output of the first group of convolution layers; providing (s708) the obtained first additional input data to a second group of one or more convolution layers, thereby generating an output of the second group of convolution layers; and encoding or decoding (s710) the video based on the output of the first group of convolution layers and the output of the second group of convolution layers, wherein the first group of convolution layers and the second group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers.

2. The method of claim 1, further comprising: combining the output of the first group of convolution layers and the output of the second group of convolution layers, thereby generating a combined output; providing the combined output to a third group of one or more convolution layers, thereby generating an output of the third group of convolution layers, wherein the video is encoded or decoded based on the output of the third group of convolution layers.

3. The method of claim 1 or 2, wherein the first additional input data is any one of values of components of predicted samples, partitional information indicating how components of samples are partitioned, block boundary strength information indicating strength of a filtering applied to a boundary of components of samples, values of quantization parameters, block size information indicating a block size, motion vector information indicating motion vectors, prediction mode information indicating a prediction mode, and/or block type information indicating a block type.

4. The method of any one of claims 1-2, wherein a number of channels included in the first group of convolution layers is more than twice of a number of channels included in the second group of convolution layers.

5. The method of claim 4, wherein the number of channels included in the first group of convolution layers is 192, and the number of channels included in the second group of convolution layers is 48, 24, or 12.

6. The method of any one of claims 1-5, the method comprising: obtaining second additional input data; and providing the second additional input data to a fourth group of one or more convolution layers, thereby generating an output of the fourth group of convolution layers, wherein the video is encoded or decoded based on the output of the fourth group of convolution layers, the first group of convolution layers and the fourth group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers, and the second group of convolution layers and the fourth group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers.

7. The method of claim 6, when dependent on claim 5, wherein the number of channels included in the second group of convolution layers is 48 or 24, and a number of channels included in the fourth group of convolution layers is 24 or 12.

8. The method of any one of claims 1-7, wherein a size of a kernel filter for the first group of convolution layers is different from a size of a kernel filter for the second group of convolution layers.

9. The method of claim 8, wherein the size of the kernel filter for first group of convolution layers is 5 x 5, and the size of the kernel filter for the second group of convolution layers is 3 x 3.

10. The method of any one of claims 1-9, wherein a convolution layer of the first group of convolution layers comprises a set of channels, a first subset of channels included in the set of channels has a kernel filter having a first size, a second subset of channels included in the set of channels has a kernel filter having a second size, and the first size and the second size are different.

11. The method of claim 10, wherein the size of the kernel filter of the first subset of channels is 5 x 5, and the size of the kernel filter of the second subset of channels is 3 x 3.

12. The method of any one of claims 1-11, wherein the first group of convolution layers comprises a first number of convolution layers, the second group of convolution layers comprises a second number of convolution layers, and the first number is greater than the second number.

13. The method of claim 12, the first number is 2 and the second number is 1.

14. The method of claim 12 or 13, wherein the first group of convolution layers includes a first convolution layer and a second convolution layer, the first convolution layer has a kernel filter having a first size, the second convolution layer has a kernel filter having a second size, and the first size and the second size are different.

15. The method of claim 14, wherein the first size is 5 x 5, and the second size is 3 x 3.

16. The method of any one of claims 1-15, wherein the first group of convolution layers has a kernel filter having a first plurality of filter values, the second group of convolution layers has a kernel filter having a second plurality of filter values, and a number of bits of each filter value included in the first plurality of filter values is greater than a number of bits of each filter value included in the second plurality of filter values.

17. The method of claim 16, wherein the number of bits of each filter value included in the first plurality of filter values is 14 bits, and the number of bits of each filter value included in the second plurality of filter values is 7 bits.

18. The method of any one of embodiments 1-17, wherein the reconstructed samples are one of reconstructed luma samples and reconstructed chroma samples, the method comprises: obtaining values of components of another one of the reconstructed luma samples and the reconstructed chroma samples; providing to a fifth group of one or more convolution layers the obtained values of the components of said another one of the reconstructed luma samples and the reconstructed chroma samples; and encoding or decoding the video based on the output of the fifth group of convolution layers, and the first group of convolution layers and the fifth group of convolution layers are different with respect to a number of channels, a size of a kernel filter, and/or a number of convolution layers.

19. A computer program (800) comprising instructions (844) which when executed by processing circuitry (802) cause the processing circuitry to perform the method of any one of claims 1-18.

20. A carrier containing the computer program of claim 19, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

21. An apparatus (800) for encoding or decoding a video, the apparatus being configured to perform the method of any one of claims 1-18.