KR20240013045A - Video In-loop Filter Adaptive to Various Noise and Characteristics - Google Patents

Abstract

This embodiment discloses a video coding method and apparatus using an in-loop filter that is adaptive to various video noises and characteristics. In this embodiment, the image decoding device generates an output block by inputting a restored block for the current block into a deep learning-based in-loop filter. The video decoding device selects a block for retraining the in-loop filter from the restoration block and then retrains the in-loop filter using the selected restoration block and the corresponding output block.

Description

Video In-loop Filter Adaptive to Various Noise and Characteristics}

This disclosure relates to a video coding method and apparatus using an in-loop filter that is adaptive to various video noises and characteristics.

The content described below simply provides background information related to this embodiment and does not constitute prior art.

Since video data has a larger amount of data than audio data or still image data, it requires a lot of hardware resources, including memory, to store or transmit it without processing for compression.

Therefore, typically, when storing or transmitting video data, the encoder compresses the video data and stores or transmits it, and the decoder receives the compressed video data, decompresses it, and plays it. These video compression technologies include H.264/AVC, HEVC (High Efficiency Video Coding), and VVC (Versatile Video Coding), which improves coding efficiency by about 30% or more compared to HEVC.

However, the size, resolution, and frame rate of the image are gradually increasing, and the amount of data that needs to be encoded is also increasing accordingly, so a new compression technology with better coding efficiency and higher picture quality improvement effect than the existing compression technology is required.

Recently, deep learning-based image processing technology is being applied to existing coding element technology. Among existing encoding technologies, coding efficiency can be improved by applying deep learning-based image processing technology to compression technologies such as inter prediction, intra prediction, in-loop filter, and transformation. Representative application examples include inter prediction based on a virtual reference frame generated based on deep learning and in-loop filter based on a noise removal model.

Meanwhile, the noise removal model-based in-loop filter can provide excellent compression noise removal performance because it is trained under the assumption that the image quality of input images in learning and real environments is similar. However, when general supervised learning is applied to a noise removal model, the performance of the in-loop filter may decrease as the difference between training image quality and verification image quality increases. Therefore, in order to improve video coding efficiency and video quality, it is necessary to consider a method of adaptively removing compression noise on the decoder side for videos with different picture quality and characteristics.

The purpose of the present disclosure is to provide a video coding method and device for decoding video using a pre-trained deep learning-based in-loop filter. At this time, the video coding method and device retrains the in-loop filter on the decoder side using video samples selected during the video decoding process, and then restores the video using the retrained in-loop filter.

According to an embodiment of the present disclosure, a method of restoring a current block performed by a video decoding apparatus includes: generating a restored block for the current block from a bitstream; Inputting the restored block into a deep learning-based in-loop filter to generate an output block, wherein the in-loop filter is pre-trained to generate an output close to the original block from the restored block; selecting a block for retraining the in-loop filter from the restoration block; and retraining the in-loop filter using the selected restoration block and the target block.

According to another embodiment of the present disclosure, a method of encoding a current block performed by an image encoding device includes: generating a reconstructed block for the current block; Generating a first output block by inputting the restored block into a deep learning-based in-loop filter, wherein the in-loop filter is trained in advance to generate an output close to the original block from the restored block; selecting a block for retraining the in-loop filter from the restoration block; Retraining the in-loop filter using the selected restoration block and target block; and generating a second output block by inputting the selected restored block into the retrained in-loop filter.

According to another embodiment of the present disclosure, a computer-readable recording medium stores a bitstream generated by an image encoding method, the image encoding method comprising: generating a restored block for a current block; Generating a first output block by inputting the restored block into a deep learning-based in-loop filter, wherein the in-loop filter is trained in advance to generate an output close to the original block from the restored block; selecting a block for retraining the in-loop filter from the restoration block; retraining the in-loop filter using the selected restoration block and target block; and inputting the selected restored block to the retrained in-loop filter to generate a second output block.

As described above, according to this embodiment, a deep learning-based in-loop filter is retrained on the decoder side using video samples selected during the video decoding process, and then the video is restored using the retrained in-loop filter. By providing a video coding method and device, it is possible to improve video coding efficiency and video quality.

1 is an example block diagram of a video encoding device that can implement the techniques of the present disclosure.
Figure 2 is a diagram to explain a method of dividing a block using the QTBTTT structure.
3A and 3B are diagrams showing a plurality of intra prediction modes including wide-angle intra prediction modes.
Figure 4 is an example diagram of neighboring blocks of the current block.
Figure 5 is an example block diagram of a video decoding device that can implement the techniques of the present disclosure.
Figure 6 is an example diagram showing the operation of a convolutional layer.
Figure 7 is an example diagram showing a Single Image Super Resolution (SISR) network.
Figure 8 is an example diagram showing a residual block used in SISR.
Figure 9 is an example diagram showing a CNN (Convolutional Neural Network) based fixed coefficient in-loop filter.
Figure 10 is an example diagram showing the form of an Adaptive Loop Filter (ALF).
Figure 11 is a block diagram showing a video decoding device according to an embodiment of the present disclosure.
Figure 12 is an example diagram showing the operation of a deep learning-based in-loop filter according to an embodiment of the present disclosure.
Figure 13 is an exemplary diagram showing retraining of a deep learning module according to an embodiment of the present disclosure.
Figure 14 is an example diagram showing retraining of a deep learning module according to another embodiment of the present disclosure.
FIG. 15 is a flowchart showing a method of encoding a current block performed by a video encoding device according to an embodiment of the present disclosure.
FIG. 16 is a flowchart showing a method for restoring a current block performed by an image decoding device according to an embodiment of the present disclosure.

Hereinafter, some embodiments of the present disclosure will be described in detail with reference to illustrative drawings. When adding reference numerals to components in each drawing, it should be noted that identical components are given the same reference numerals as much as possible even if they are shown in different drawings. Additionally, in describing the present embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present embodiments, the detailed description will be omitted.

1 is an example block diagram of a video encoding device that can implement the techniques of the present disclosure. Hereinafter, the video encoding device and its sub-configurations will be described with reference to the illustration in FIG. 1.

The image encoding device includes a picture division unit 110, a prediction unit 120, a subtractor 130, a transform unit 140, a quantization unit 145, a rearrangement unit 150, an entropy encoding unit 155, and an inverse quantization unit. It may be configured to include (160), an inverse transform unit (165), an adder (170), a loop filter unit (180), and a memory (190).

Each component of the video encoding device may be implemented as hardware or software, or may be implemented as a combination of hardware and software. Additionally, the function of each component may be implemented as software and a microprocessor may be implemented to execute the function of the software corresponding to each component.

One image (video) consists of one or more sequences including a plurality of pictures. Each picture is divided into a plurality of regions and encoding is performed for each region. For example, one picture is divided into one or more tiles and/or slices. Here, one or more tiles can be defined as a tile group. Each tile or/slice is divided into one or more Coding Tree Units (CTUs). And each CTU is divided into one or more CUs (Coding Units) by a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in one CTU is encoded as the syntax of the CTU. Additionally, information commonly applied to all blocks within one slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more pictures is a picture parameter set (PPS) or picture parameter set. Encoded in the header. Furthermore, information commonly referenced by multiple pictures is encoded in a sequence parameter set (SPS). And, information commonly referenced by one or more SPSs is encoded in a video parameter set (VPS). Additionally, information commonly applied to one tile or tile group may be encoded as the syntax of a tile or tile group header. Syntax included in the SPS, PPS, slice header, tile, or tile group header may be referred to as high level syntax.

The picture division unit 110 determines the size of the CTU. Information about the size of the CTU (CTU size) is encoded as SPS or PPS syntax and transmitted to the video decoding device.

The picture division unit 110 divides each picture constituting the image into a plurality of CTUs with a predetermined size and then recursively divides the CTUs using a tree structure. . The leaf node in the tree structure becomes the CU, the basic unit of encoding.

The tree structure is QuadTree (QT), in which the parent node is divided into four child nodes (or child nodes) of the same size, or BinaryTree, in which the parent node is divided into two child nodes. , BT), or a TernaryTree (TT) in which the parent node is divided into three child nodes in a 1:2:1 ratio, or a structure that mixes two or more of these QT structures, BT structures, and TT structures. there is. For example, a QuadTree plus BinaryTree (QTBT) structure may be used, or a QuadTree plus BinaryTree TernaryTree (QTBTTT) structure may be used. Here, BTTT may be combined and referred to as MTT (Multiple-Type Tree).

Figure 2 is a diagram to explain a method of dividing a block using the QTBTTT structure.

As shown in Figure 2, the CTU can first be divided into a QT structure. Quadtree splitting can be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of the leaf node allowed in QT. The first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoder 155 and signaled to the image decoding device. If the leaf node of QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in BT, it may be further divided into either the BT structure or the TT structure. In the BT structure and/or TT structure, there may be multiple division directions. For example, there may be two directions in which the block of the node is divided: horizontally and vertically. As shown in Figure 2, when MTT splitting begins, a second flag (mtt_split_flag) indicates whether the nodes have been split, and if split, an additional flag indicating the splitting direction (vertical or horizontal) and/or the splitting type (Binary). Or, a flag indicating Ternary) is encoded by the entropy encoding unit 155 and signaled to the video decoding device.

Alternatively, prior to encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of the lower layer, a CU split flag (split_cu_flag) indicating whether the node is split is encoded. It could be. If the CU split flag (split_cu_flag) value indicates that it is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU (coding unit), which is the basic unit of coding. When the CU split flag (split_cu_flag) value indicates splitting, the video encoding device starts encoding from the first flag in the above-described manner.

When QTBT is used as another example of a tree structure, there are two types: a type that horizontally splits the block of the node into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that splits it vertically (i.e., symmetric vertical splitting). Branches may exist. A split flag (split_flag) indicating whether each node of the BT structure is divided into blocks of a lower layer and split type information indicating the type of division are encoded by the entropy encoder 155 and transmitted to the video decoding device. Meanwhile, there may be an additional type that divides the block of the corresponding node into two asymmetric blocks. The asymmetric form may include dividing the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or may include dividing the block of the corresponding node diagonally.

A CU can have various sizes depending on the QTBT or QTBTTT division from the CTU. Hereinafter, the block corresponding to the CU (i.e., leaf node of QTBTTT) to be encoded or decoded is referred to as the 'current block'. Depending on the adoption of QTBTTT partitioning, the shape of the current block may be rectangular as well as square.

The prediction unit 120 predicts the current block and generates a prediction block. The prediction unit 120 includes an intra prediction unit 122 and an inter prediction unit 124.

In general, each current block in a picture can be coded predictively. Typically, prediction of the current block is done using intra prediction techniques (using data from the picture containing the current block) or inter prediction techniques (using data from pictures coded before the picture containing the current block). It can be done. Inter prediction includes both one-way prediction and two-way prediction.

The intra prediction unit 122 predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture including the current block. There are multiple intra prediction modes depending on the prediction direction. For example, as shown in FIG. 3A, the plurality of intra prediction modes may include two non-directional modes including a planar mode and a DC mode and 65 directional modes. The surrounding pixels and calculation formulas to be used are defined differently for each prediction mode.

For efficient directional prediction of the rectangular-shaped current block, the directional modes (67 to 80, -1 to -14 intra prediction modes) shown by dotted arrows in FIG. 3B can be additionally used. These may be referred to as “wide angle intra-prediction modes”. In Figure 3b, the arrows point to corresponding reference samples used for prediction and do not indicate the direction of prediction. The predicted direction is opposite to the direction indicated by the arrow. Wide-angle intra prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without transmitting additional bits when the current block is rectangular. At this time, among the wide-angle intra prediction modes, some wide-angle intra prediction modes available for the current block may be determined according to the ratio of the width and height of the rectangular current block. For example, wide-angle intra prediction modes with angles smaller than 45 degrees (intra prediction modes 67 to 80) are available when the current block is in the form of a rectangle whose height is smaller than its width, and wide-angle intra prediction modes with angles larger than -135 degrees are available. Intra prediction modes (-1 to -14 intra prediction modes) are available when the current block has a rectangular shape with a width greater than the height.

The intra prediction unit 122 can determine the intra prediction mode to be used to encode the current block. In some examples, intra prediction unit 122 may encode the current block using multiple intra prediction modes and select an appropriate intra prediction mode to use from the tested modes. For example, the intra prediction unit 122 calculates rate-distortion values using rate-distortion analysis for several tested intra-prediction modes and has the best rate-distortion characteristics among the tested modes. You can also select intra prediction mode.

The intra prediction unit 122 selects one intra prediction mode from a plurality of intra prediction modes and predicts the current block using surrounding pixels (reference pixels) and an operation formula determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoding unit 155 and transmitted to the video decoding device.

The inter prediction unit 124 generates a prediction block for the current block using a motion compensation process. The inter prediction unit 124 searches for a block most similar to the current block in a reference picture that has been encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector (MV) corresponding to the displacement between the current block in the current picture and the prediction block in the reference picture is generated. Typically, motion estimation is performed on the luma component, and a motion vector calculated based on the luma component is used for both the luma component and the chroma component. Motion information including information about the reference picture and information about the motion vector used to predict the current block is encoded by the entropy encoding unit 155 and transmitted to the video decoding device.

The inter prediction unit 124 may perform interpolation on a reference picture or reference block to increase prediction accuracy. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. If the process of searching for the block most similar to the current block is performed for the interpolated reference picture, the motion vector can be expressed with precision in decimal units rather than precision in integer samples. The precision or resolution of the motion vector may be set differently for each target area to be encoded, for example, slice, tile, CTU, CU, etc. When such adaptive motion vector resolution (AMVR) is applied, information about the motion vector resolution to be applied to each target area must be signaled for each target area. For example, if the target area is a CU, information about the motion vector resolution applied to each CU is signaled. Information about motion vector resolution may be information indicating the precision of a differential motion vector, which will be described later.

Meanwhile, the inter prediction unit 124 may perform inter prediction using bi-prediction. In the case of bidirectional prediction, two reference pictures and two motion vectors indicating the positions of blocks most similar to the current block within each reference picture are used. The inter prediction unit 124 selects the first reference picture and the second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block within each reference picture. Create a first reference block and a second reference block. Then, the first reference block and the second reference block are averaged or weighted to generate a prediction block for the current block. Then, motion information including information about the two reference pictures used to predict the current block and information about the two motion vectors is transmitted to the entropy encoding unit 155. Here, reference picture list 0 may be composed of pictures before the current picture in display order among the restored pictures, and reference picture list 1 may be composed of pictures after the current picture in display order among the restored pictures. there is. However, it is not necessarily limited to this, and in terms of display order, relief pictures after the current picture may be additionally included in reference picture list 0, and conversely, relief pictures before the current picture may be additionally included in reference picture list 1. may be included.

Various methods can be used to minimize the amount of bits required to encode motion information.

For example, if the reference picture and motion vector of the current block are the same as the reference picture and motion vector of the neighboring block, the motion information of the current block can be transmitted to the video decoding device by encoding information that can identify the neighboring block. This method is called ‘merge mode’.

In the merge mode, the inter prediction unit 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from neighboring blocks of the current block.

As shown in FIG. 4, the surrounding blocks for deriving merge candidates include the left block (A0), bottom left block (A1), top block (B0), and top right block (B1) adjacent to the current block in the current picture. ), and all or part of the upper left block (B2) can be used. Additionally, a block located within a reference picture (which may be the same or different from the reference picture used to predict the current block) rather than the current picture where the current block is located may be used as a merge candidate. For example, a block co-located with the current block within the reference picture or blocks adjacent to the co-located block may be additionally used as merge candidates. If the number of merge candidates selected by the method described above is less than the preset number, the 0 vector is added to the merge candidates.

The inter prediction unit 124 uses these neighboring blocks to construct a merge list including a predetermined number of merge candidates. A merge candidate to be used as motion information of the current block is selected from among the merge candidates included in the merge list, and merge index information is generated to identify the selected candidate. The generated merge index information is encoded by the entropy encoding unit 155 and transmitted to the video decoding device.

Merge skip mode is a special case of merge mode. After performing quantization, when all transformation coefficients for entropy encoding are close to zero, only peripheral block selection information is transmitted without transmitting residual signals. By using merge skip mode, relatively high coding efficiency can be achieved in low-motion images, still images, screen content images, etc.

Hereinafter, merge mode and merge skip mode are collectively referred to as merge/skip mode.

Another method for encoding motion information is AMVP (Advanced Motion Vector Prediction) mode.

In AMVP mode, the inter prediction unit 124 uses neighboring blocks of the current block to derive predicted motion vector candidates for the motion vector of the current block. The surrounding blocks used to derive predicted motion vector candidates include the left block (A0), bottom left block (A1), top block (B0), and top right block adjacent to the current block in the current picture shown in FIG. All or part of B1), and the upper left block (B2) can be used. Additionally, a block located within a reference picture (which may be the same or different from the reference picture used to predict the current block) rather than the current picture where the current block is located will be used as a surrounding block used to derive prediction motion vector candidates. It may be possible. For example, a collocated block located at the same location as the current block within the reference picture or blocks adjacent to the block at the same location may be used. If the number of motion vector candidates is less than the preset number by the method described above, the 0 vector is added to the motion vector candidates.

The inter prediction unit 124 derives predicted motion vector candidates using the motion vectors of the neighboring blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, the predicted motion vector is subtracted from the motion vector of the current block to calculate the differential motion vector.

The predicted motion vector can be obtained by applying a predefined function (eg, median, average value calculation, etc.) to the predicted motion vector candidates. In this case, the video decoding device also knows the predefined function. In addition, since the neighboring blocks used to derive predicted motion vector candidates are blocks for which encoding and decoding have already been completed, the video decoding device also already knows the motion vectors of the neighboring blocks. Therefore, the video encoding device does not need to encode information to identify the predicted motion vector candidate. Therefore, in this case, information about the differential motion vector and information about the reference picture used to predict the current block are encoded.

Meanwhile, the predicted motion vector may be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected prediction motion vector candidate is additionally encoded, along with information about the differential motion vector and information about the reference picture used to predict the current block.

The subtractor 130 generates a residual block by subtracting the prediction block generated by the intra prediction unit 122 or the inter prediction unit 124 from the current block.

The transform unit 140 converts the residual signals in the residual block having pixel values in the spatial domain into transform coefficients in the frequency domain. The conversion unit 140 may convert the residual signals in the residual block by using the entire size of the residual block as a conversion unit, or divide the residual block into a plurality of subblocks and perform conversion by using the subblocks as a conversion unit. You may. Alternatively, the residual signals can be converted by dividing them into two subblocks, a transform area and a non-transformation region, and using only the transform region subblock as a transform unit. Here, the transformation area subblock may be one of two rectangular blocks with a size ratio of 1:1 based on the horizontal axis (or vertical axis). In this case, a flag indicating that only the subblock has been converted (cu_sbt_flag), directional (vertical/horizontal) information (cu_sbt_horizontal_flag), and/or position information (cu_sbt_pos_flag) are encoded by the entropy encoding unit 155 and signaled to the video decoding device. do. In addition, the size of the transform area subblock may have a size ratio of 1:3 based on the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) that distinguishes the corresponding division is additionally encoded by the entropy encoding unit 155 to encode the image. Signaled to the decryption device.

Meanwhile, the transformation unit 140 can separately perform transformation on the residual block in the horizontal and vertical directions. For transformation, various types of transformation functions or transformation matrices can be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as MTS (Multiple Transform Set). The conversion unit 140 may select a conversion function pair with the best conversion efficiency among MTSs and convert the residual blocks in the horizontal and vertical directions, respectively. Information (mts_idx) about the transformation function pair selected from the MTS is encoded by the entropy encoder 155 and signaled to the video decoding device.

The quantization unit 145 quantizes the transform coefficients output from the transform unit 140 using a quantization parameter, and outputs the quantized transform coefficients to the entropy encoding unit 155. The quantization unit 145 may directly quantize a residual block related to a certain block or frame without conversion. The quantization unit 145 may apply different quantization coefficients (scaling values) depending on the positions of the transform coefficients within the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to the video decoding device.

The rearrangement unit 150 may rearrange coefficient values for the quantized residual values.

The rearrangement unit 150 can change a two-dimensional coefficient array into a one-dimensional coefficient sequence using coefficient scanning. For example, the realignment unit 150 can scan from DC coefficients to coefficients in the high frequency region using zig-zag scan or diagonal scan to output a one-dimensional coefficient sequence. . Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans a two-dimensional coefficient array in the column direction or a horizontal scan that scans the two-dimensional block-type coefficients in the row direction may be used instead of the zig-zag scan. That is, the scan method to be used among zig-zag scan, diagonal scan, vertical scan, and horizontal scan may be determined depending on the size of the transformation unit and the intra prediction mode.

The entropy encoding unit 155 uses various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb to encode the one-dimensional quantized transform coefficients output from the reordering unit 150. A bitstream is created by encoding the sequence.

In addition, the entropy encoder 155 encodes information such as CTU size, CU split flag, QT split flag, MTT split type, and MTT split direction related to block splitting, so that the video decoding device can encode blocks in the same way as the video coding device. Allow it to be divided. In addition, the entropy encoding unit 155 encodes information about the prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and generates intra prediction information (i.e., intra prediction) according to the prediction type. Information about the mode) or inter prediction information (coding mode of motion information (merge mode or AMVP mode), merge index in case of merge mode, information on reference picture index and differential motion vector in case of AMVP mode) is encoded. Additionally, the entropy encoding unit 155 encodes information related to quantization, that is, information about quantization parameters and information about the quantization matrix.

The inverse quantization unit 160 inversely quantizes the quantized transform coefficients output from the quantization unit 145 to generate transform coefficients. The inverse transform unit 165 restores the residual block by converting the transform coefficients output from the inverse quantization unit 160 from the frequency domain to the spatial domain.

The adder 170 restores the current block by adding the restored residual block and the prediction block generated by the prediction unit 120. Pixels in the restored current block are used as reference pixels when intra-predicting the next block.

The loop filter unit 180 restores pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation/quantization. Perform filtering on them. The loop filter unit 180 is an in-loop filter and may include all or part of a deblocking filter 182, a Sample Adaptive Offset (SAO) filter 184, and an Adaptive Loop Filter (ALF) 186. there is.

The deblocking filter 182 filters the boundaries between restored blocks to remove blocking artifacts caused by block-level encoding/decoding, and the SAO filter 184 and ALF 186 perform deblocking filtering. Additional filtering is performed on the image. The SAO filter 184 and the ALF 186 are filters used to compensate for differences between restored pixels and original pixels caused by lossy coding. The SAO filter 184 improves not only subjective image quality but also coding efficiency by applying an offset in units of CTU. In comparison, the ALF 186 performs filtering on a block basis, distinguishing the edge and degree of change of the block and applying different filters to compensate for distortion. Information about filter coefficients to be used in ALF may be encoded and signaled to a video decoding device.

The restored block filtered through the deblocking filter 182, SAO filter 184, and ALF 186 is stored in the memory 190. When all blocks in one picture are reconstructed, the reconstructed picture can be used as a reference picture for inter prediction of blocks in the picture to be encoded later.

The video encoding device can store the bitstream of the encoded video data in a non-transitory recording medium or transmit it to the video decoding device using a communication network.

Figure 5 is an example block diagram of a video decoding device that can implement the techniques of the present disclosure. Hereinafter, the video decoding device and its sub-configurations will be described with reference to FIG. 5.

The image decoding device includes an entropy decoding unit 510, a rearrangement unit 515, an inverse quantization unit 520, an inverse transform unit 530, a prediction unit 540, an adder 550, a loop filter unit 560, and a memory ( 570).

Like the video encoding device of FIG. 1, each component of the video decoding device may be implemented as hardware or software, or may be implemented as a combination of hardware and software. Additionally, the function of each component may be implemented as software and a microprocessor may be implemented to execute the function of the software corresponding to each component.

The entropy decoder 510 decodes the bitstream generated by the video encoding device, extracts information related to block division, determines the current block to be decoded, and provides prediction information and residual signals needed to restore the current block. Extract information about

The entropy decoder 510 extracts information about the CTU size from a Sequence Parameter Set (SPS) or Picture Parameter Set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the CTU is determined as the highest layer of the tree structure, that is, the root node, and the CTU is divided using the tree structure by extracting the division information for the CTU.

For example, when dividing a CTU using the QTBTTT structure, first extract the first flag (QT_split_flag) related to the division of the QT and split each node into four nodes of the lower layer. And, for the node corresponding to the leaf node of QT, the second flag (mtt_split_flag) and split direction (vertical / horizontal) and/or split type (binary / ternary) information related to the split of MTT are extracted and the leaf node is divided into MTT. Divide by structure. Accordingly, each node below the leaf node of QT is recursively divided into a BT or TT structure.

As another example, when splitting a CTU using the QTBTTT structure, first extract the CU split flag (split_cu_flag) indicating whether to split the CU, and if the corresponding block is split, extract the first flag (QT_split_flag). It may be possible. During the splitting process, each node may undergo zero or more repetitive MTT splits after zero or more repetitive QT splits. For example, MTT division may occur immediately in the CTU, or conversely, only multiple QT divisions may occur.

As another example, when dividing a CTU using the QTBT structure, the first flag (QT_split_flag) related to the division of the QT is extracted and each node is divided into four nodes of the lower layer. And, for the node corresponding to the leaf node of QT, a split flag (split_flag) indicating whether to further split into BT and split direction information are extracted.

Meanwhile, when the entropy decoding unit 510 determines the current block to be decoded using division of the tree structure, it extracts information about the prediction type indicating whether the current block is intra-predicted or inter-predicted. When prediction type information indicates intra prediction, the entropy decoder 510 extracts syntax elements for intra prediction information (intra prediction mode) of the current block. When prediction type information indicates inter prediction, the entropy decoder 510 extracts syntax elements for inter prediction information, that is, information indicating a motion vector and a reference picture to which the motion vector refers.

Additionally, the entropy decoding unit 510 extracts information about quantized transform coefficients of the current block as quantization-related information and information about residual signals.

The reordering unit 515 re-organizes the sequence of one-dimensional quantized transform coefficients entropy decoded in the entropy decoding unit 510 into a two-dimensional coefficient array (i.e., in reverse order of the coefficient scanning order performed by the image encoding device). block).

The inverse quantization unit 520 inversely quantizes the quantized transform coefficients and inversely quantizes the quantized transform coefficients using a quantization parameter. The inverse quantization unit 520 may apply different quantization coefficients (scaling values) to quantized transform coefficients arranged in two dimensions. The inverse quantization unit 520 may perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transform coefficients.

The inverse transform unit 530 inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore the residual signals, thereby generating a residual block for the current block.

In addition, when the inverse transformation unit 530 inversely transforms only a partial area (subblock) of the transformation block, a flag (cu_sbt_flag) indicating that only the subblock of the transformation block has been transformed, and directionality (vertical/horizontal) information of the subblock (cu_sbt_horizontal_flag) ) and/or extracting the position information (cu_sbt_pos_flag) of the subblock, and inversely transforming the transformation coefficients of the corresponding subblock from the frequency domain to the spatial domain to restore the residual signals, and for the area that has not been inversely transformed, the residual signals are set to “0”. By filling in the values, the final residual block for the current block is created.

In addition, when MTS is applied, the inverse transform unit 530 determines a transformation function or transformation matrix to be applied in the horizontal and vertical directions, respectively, using the MTS information (mts_idx) signaled from the video encoding device, and uses the determined transformation function. Inverse transformation is performed on the transformation coefficients in the transformation block in the horizontal and vertical directions.

The prediction unit 540 may include an intra prediction unit 542 and an inter prediction unit 544. The intra prediction unit 542 is activated when the prediction type of the current block is intra prediction, and the inter prediction unit 544 is activated when the prediction type of the current block is inter prediction.

The intra prediction unit 542 determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoder 510, and provides a reference around the current block according to the intra prediction mode. Predict the current block using pixels.

The inter prediction unit 544 uses the syntax elements for the inter prediction mode extracted from the entropy decoder 510 to determine the motion vector of the current block and the reference picture to which the motion vector refers, and uses the motion vector and the reference picture to determine the motion vector of the current block. Use it to predict the current block.

The adder 550 restores the current block by adding the residual block output from the inverse transform unit 530 and the prediction block output from the inter prediction unit 544 or intra prediction unit 542. Pixels in the restored current block are used as reference pixels when intra-predicting a block to be decoded later.

The loop filter unit 560 may include a deblocking filter 562, a SAO filter 564, and an ALF 566 as an in-loop filter. The deblocking filter 562 performs deblocking filtering on the boundaries between restored blocks to remove blocking artifacts that occur due to block-level decoding. The SAO filter 564 and the ALF 566 perform additional filtering on the reconstructed block after deblocking filtering to compensate for the difference between the reconstructed pixels and the original pixels caused by lossy coding. do. The filter coefficient of ALF is determined using information about the filter coefficient decoded from the non-stream.

The restored block filtered through the deblocking filter 562, SAO filter 564, and ALF 566 is stored in the memory 570. When all blocks in one picture are reconstructed, the reconstructed picture is later used as a reference picture for inter prediction of blocks in the picture to be encoded.

This embodiment relates to encoding and decoding of images (videos) as described above. More specifically, a video coding method and device for decoding video using a pre-trained deep learning-based in-loop filter are provided. At this time, the video coding method and device retrains the in-loop filter on the decoder side using video samples selected during the video decoding process, and then decodes the video using the retrained in-loop filter.

The following embodiments may be performed by various components within a video encoding device. Additionally, it can be performed by several components within a video decoding device.

The video encoding device may generate signaling information related to this embodiment in terms of bit rate distortion optimization when encoding the current block. The video encoding device can encode the video using the entropy encoding unit 155 and then transmit it to the video decoding device. The video decoding device can decode signaling information related to decoding the current block from the bitstream using the entropy decoding unit 510.

In the following description, the term 'target block' may be used with the same meaning as a current block or a coding unit (CU), or may mean a partial area of a coding unit.

Additionally, the fact that the value of one flag is true indicates that the flag is set to 1. Additionally, the value of one flag being false indicates a case where the flag is set to 0.

I. CNN(Convolutional Neural Network)

CNN refers to a neural network consisting of multiple convolution layers and pooling layers, and is a deep learning technology known to be most suitable for image processing. The convolution layer extracts a feature map (used interchangeably with 'feature') using multiple kernels or filters. At this time, the kernel coefficient that makes up the filter is a parameter determined during the learning process.

Among CNN's convolutional layers, the front layer close to the input extracts a feature map that responds to simple, low-level image features such as lines, points, or faces, and the back layer close to the output extracts texture, Extract feature maps that respond to higher levels, such as object parts.

Figure 6 is an example diagram showing the operation of a convolutional layer according to an embodiment of the present disclosure.

The convolution layer creates a feature map from the input image using a convolution operation. In the example of Figure 6, a kernel (or filter) with a kernel size of 3×3 is represented. Kernel size is also referred to as kernel size or filter size. The kernel has kernel parameters (kernel parameters or filter parameters), also called weights. The kernel illustrated in Figure 6 has a total of 9 kernel parameters. Kernel parameters are initially set to random values, and the values can be updated based on learning.

The convolution layer performs a convolution operation using blocks equal to the kernel size in the input image. At this time, a block as large as the kernel size in the input image is referred to as a window.

When filtering an input image is performed in raster-scan order, the window movement size is called stride. In the example of Figure 6, the stride is 1. If the stride is set to 2, the convolution operation is performed by spacing the window by 2 samples, and as a result, the horizontal and vertical sizes of the feature map become half of the horizontal and vertical sizes of the input image.

As described above, one convolutional layer may include multiple filters. The number of filters (number of filters/filters) or the number of kernels (number of kernels/kernels) is called a channel. That is, the number of channels is equal to the number of filters. Additionally, the number of filters determines the size of the dimension of the feature map.

Padding refers to a method of expanding input data by filling the area around the input data with a specific value before performing a convolution operation. Padding is mainly used to control the spatial size of output data. The value used for padding can be determined by hyperparameters, but zero-padding is mainly used. If padding is not used, the spatial size of the output data decreases every time it passes through a convolutional layer, which may cause the problem of boundary information disappearing. Therefore, to prevent this problem, padding is used. In other words, padding can be used to make the spatial sizes of the output data and input data of the convolutional layer the same.

The deconvolution layer performs the opposite operation to the convolution layer. The deconvolution layer generates the desired data image as output from the input feature map.

The pooling layer performs pooling, a process of sub-sampling the feature map generated by the convolution layer. The pooling layer uses a 2×2 window to select samples so that the output result is half the width and height of the input. In other words, the pooling layer is used to reduce the size of the input image or input feature map by consolidating the 2×2 area into one sample.

The opposite concept of a pooling layer is defined as an unpooling layer. The unpooling layer, as opposed to the pooling layer, plays the role of enlarging the dimension and is mainly used after the deconvolution layer.

The convolutional encoder-decoder structure is a network structure consisting of a pair of convolutional layers and deconvolution layers. The convolutional encoder consists of a convolutional layer and a pooling layer and outputs a feature map (or feature vector) from the input image. The final output vector of the convolutional encoder is also referred to as a latent vector. The convolutional decoder consists of a deconvolution layer and an unpooling layer and generates an output image from a feature map or latent vector.

The input and output of the convolutional encoder-decoder can be set in various ways depending on the purpose of the application and network. For example, the input and output may be an optical flow map, saliency map, image frame, etc.

Figure 7 is an example diagram showing a SISR network.

An example of CNN application is SISR (Single Image Super Resolution). SISR networks generate high-resolution images as output from low-resolution input images. A SISR network may include multiple convolutional layers, as illustrated in FIG. 7. Each convolutional layer includes an activation function such as ReLU (Rectified Linear Unit). The parameters of the SISR network can be trained so that the generated Super Resolution (SR) image is close to the Ground Truth (GT).

SR methods using CNN can improve SR performance by increasing depth (for example, this means increasing the number of convolutional layers). To overcome the overfitting problem in learning that may occur as the depth increases, a residual block that can perform skip connection and residual learning is installed in the SISR network. It can be used. The residual block includes a skip path in addition to the path that applies the convolution operation to the input feature x _l , as illustrated in FIG. 8. Additionally, when generating output x _l+1 , the residual block may select a path to which a convolution operation is applied or a skip path based on learning efficiency. In the example of Figure 8, the residual block includes a BN (Batch Normalization) layer.

As an example, EDSR (Enhanced Deep residual networks for SISR) increases the performance of the network by sequentially connecting residual blocks to increase depth. As another example, VDSR (Accurate Image Super-Resolution Using Very Deep Convolutional Networks) is a CNN model based on the VGG (Visual Geometry Group) network and uses residual learning, a method of adding residual frames to the final output. use. VDSR adds a residual signal to the end of the network, adding the residual signal to the input signal.

As another example, CNN can be used as an in-loop filter in a video encoding device or video decoding device. At this time, the deep learning-based in-loop filter can be applied to any position in the existing loop filter units 180 and 560 consisting of a deblocking filter, SAO filter, and ALF.

As a deep learning-based in-loop filter, there is a fixed coefficient in-loop filter. The fixed coefficient deep learning in-loop filter uses the same CNN kernel parameters stored on the video encoding device and the video decoding device.

Figure 9 is an example diagram showing a CNN-based fixed coefficient in-loop filter.

The input block (or input frame) goes through a normalized QP map and is then passed on to the next step. The normalized QP map is used to reduce inference errors when quantization noise of different intensities is mixed during the learning and inference process. The kernel parameters constituting the DRU (Dense Residual Unit) and the convolution layer can be stored and used in the same way in the video encoding device and the video decoding device. In the example of FIG. 9, each DRU may include all or part of a convolutional layer, a ReLU layer, and a depth-wise separable convolutional (DSC) layer.

The fixed-coefficient deep learning in-loop filter must provide general performance for various video frames, so it has the disadvantage that the number of CNN layers becomes deeper and the computation time becomes longer.

II. Supervised and self-learning

In training deep learning networks, supervised learning is a learning method that uses label information. At this time, the process of calculating weights based on training data and labels is called training. Training can be performed using the stochastic gradient descent (SGD) algorithm, usually followed by a back propagation algorithm. The process of calculating the output in a feed-forward manner using weights, which are parameters calculated according to training, is called inference or testing.

Self-supervised learning is a learning method that uses only image information without using data label information, and trains a deep learning network using only training data without an expensive annotation process. For deep learning networks that learn feature representations (representation learning) using large-scale image data, self-learning can improve the performance of deep learning networks by applying very simple tuning to specific tasks or domains.

Contrastive learning is one of the self-learning techniques. In contrast learning, two samples are formed into a pair, and then the formed pair is used as input to a deep learning network. In contrastive learning, training is performed based on whether the two samples that make up the pair are similar. That is, if the two samples are different pairs, they are configured as negative pairs, and if the two samples are similar pairs, they are configured as positive pairs, and then the constructed pairs are used for training. Considering all negative pairs excessively increases the complexity of training, so the goal of contrastive learning is to appropriately select pairs that are helpful for training.

For example, by applying two data augmentations to a single image, two similar After implementing an image, a pair of two implemented images can be created as a positive pair. Additionally, a pair implemented by combining different images may be created as a negative pair. Contrast learning performs self-learning without any supervised learning by training a deep learning network using the InfoNCE loss function. Here, NCE stands for noise contrastive estimation. The InfoNCE loss function L _N is defined as Equation 1 based on the ratio of the similarity of positive pairs to the sum of the similarities of positive and negative pairs.

Here, exp(f(x) ^T f(x ⁺ )) is the similarity of the positive pair, and exp(f(x) ^T f(x ^- )) is the similarity of the negative pair. f(x) represents the representation of each image included in the positive or negative pair, that is, the output of the deep learning network used. Contrast learning trains a deep learning network to reduce the InfoNCE loss function by increasing the similarity of positive pairs.

III. Adaptive Loop Filter (ALF)

VVC's ALF (186, 566) uses an adaptive linear filter based on the Wiener-Hopf equation to approximate the restored video frame closely to the original. The video encoding device uses the output samples of the SAO 184 to calculate the filter coefficients of the ALF 186 according to bit rate-distortion optimization and then transmits them to the video decoding device. ALF (186, 566) is composed of a 7×7 diamond shape and a 5×5 diamond shape as shown in the example of FIG. 10 and is used for luma and chroma samples, respectively. Filter shape and size can be determined considering the balance between coding efficiency and computational complexity. For example, the computational complexity of ALF 186, 566 can be reduced using symmetric FIR filters.

To derive the filter coefficient c _i illustrated in Figure 10, a sample is used at that location. The filtered sample I _f (x,y) at the current position (x,y) can be calculated as shown in Equation 2 according to a 7-bit precision operation.

Here, N represents the number of filter coefficients. r _i is the difference value between the current sample and adjacent samples and is calculated according to Equation 3.

Here, b _i is a clipping parameter.

ALF (186, 566) uses up to 25 sets of filter coefficients for the luma component and applies them to 4×4 sub-blocks. According to the gradient information of the local block calculated using a Laplacian filter, the 4×4 sub-block is classified into one of 25 classes. Specifically, the classification index for a class is derived from a combination of five orientation attributes expressing the intensity and direction of texture components and five activity attributes of subblocks. Additionally, geometric transformations such as 90-degree rotations, diagonal transitions, and vertical transitions can be applied to the filter coefficients before filtering. By taking into account various orientations using geometric transformations, a wider variety of block characteristics can be processed using a smaller set of filter coefficients.

In addition to sub-block units, application may be determined on a CTU unit. For chroma components, up to 8 filters are used at the CTU level. Chroma ALF can be activated only when luma ALF is activated at the corresponding level.

Meanwhile, an Adaptation Parameter Set (APS) is used to convey ALF filter parameters including a set of filter coefficients. As described above, up to 25 sets of filter coefficients can be estimated for the luma component and up to 8 sets of filter coefficients for the chroma component. When using the same ALF coefficient for different slices, the index of the reference APS can be signaled instead of redundantly retransmitting.

In video applications such as High-Dynamic Range (HDR) and Wide Color Gamut (WCG), restoration of video color is very important. Cross-Component ALF (CC-ALF) uses the correlation between the current chroma sample and the luma sample at that location to modify chroma samples in parallel with ALF.

The following embodiments are described focusing on a video decoding device, but can also be performed by a video encoding device.

IV. Embodiments according to the present invention

As described above, the deep learning-based in-loop filter included in the video decoding device is trained in advance based on a large amount of learning data. After training is completed, the same deep learning module is installed in the in-loop filter of the video encoding device and the video decoding device. The deep learning module performs video compression by sufficiently removing quantization noise from general images. At this time, as described above, the deep learning module can be trained in advance to generate an output that is close to the original sample from the video sample mixed with quantization noise after compression. The deep learning module can be used in an image encoding device and an image decoding device with fixed parameters.

Hereinafter, deep learning-based in-loop filters and deep learning modules are used interchangeably. Video samples represent selected or sampled video blocks and can be used interchangeably with video blocks.

However, the filter performance of the deep learning module may depend on the characteristics of the video samples used for training. For example, if video samples with characteristics not used in the training process are input, filter performance may deteriorate. Video samples that are input during the video compression process may have various statistical characteristics depending on the strength of quantization noise and the characteristics of the sensor used to produce/generate the video. In this embodiment, in order to solve the above-described performance degradation problem, self-learning is used on the decoder side using video samples generated during the video decoding process.

Figure 11 is a block diagram showing a video decoding device according to an embodiment of the present disclosure.

The video decoding device according to this embodiment may include all or part of a video block decoding unit 1110, a video block sampling unit 1120, and a retraining unit 1130, in addition to the components illustrated in FIG. 5. You can.

The video block decoder 1110 restores video blocks from the bitstream. To restore a video block, the video block decoder 1110 may include all or part of the components illustrated in FIG. 5. The video block decoder 1110 generates a video block to be provided as an input to a deep learning-based in-loop filter.

The video block sampling unit 1120 selects a video block for retraining from among the video blocks provided from the video block decoding unit 1110 in order to apply decoder-side retraining to the deep learning module.

The retraining unit 1130 updates the parameters of the deep learning module on the decoder side using the selected video block. The video decoding device can apply the retrained deep learning module to in-loop filtering of the next block.

Although the video encoding device has original video samples, this embodiment can also be applied to the decoder side of the video encoding device. To perform the same operation as the decoder side of the video decoding device, the video encoding device may perform the same operation as the example of FIG. 11 on the decoder side. Here, the decoder side of the video encoding device includes a prediction unit 120, an inverse quantization unit 160, an inverse transform unit 165, an adder 170, a loop filter unit 180, and a memory 190 among the components of FIG. 1. may include.

Hereinafter, a method by which the video block sampling unit 1120 samples video blocks for retraining of a deep learning-based in-loop filter will be described.

The video encoding device can select a block using an in-loop filter in terms of bit rate-distortion optimization. For example, the conventional deep learning-based in-loop filter as described above is applied on a video block or slice basis. When using a deep learning-based in-loop filter, the video encoding device sets the flag indicating whether to use the in-loop filter (hereinafter referred to as 'in-loop filter flag') to 1, and then sends the in-loop filter flag to the video decoding device. transmit. If the decoded in-loop filter flag is 1, the video decoding device applies a deep learning-based in-loop filter to improve the image quality of the corresponding block. On the other hand, if the decoded in-loop filter flag is 0, the video decoding device does not apply the deep learning-based in-loop filter to the corresponding block.

VVC's ALF technology also controls whether to apply ALF (566) using a block-level or slice-level flag (hereinafter referred to as 'ALF flag'). That is, the video decoding device can determine whether to apply ALF 566 to the corresponding block according to the received flag.

As an example, when the in-loop filter flag is true, the deep learning-based in-loop filter in the video decoding device generates an improved output x _{rec_f} from the decoded sample x _rec , as shown in the example of FIG. 12. At this time, the video block sampling unit 1120 can update the parameters of the in-loop filter by using the current sample x _rec for retraining for the next input video signals. Alternatively, an improved in-loop filter may be used for the current sample as well.

As another example, the video block sampling unit 1120 may determine whether to sample the corresponding block according to parameters used when compressing the video block. For example, if the height and width of a block are smaller than a certain threshold, the block can be used for retraining. Alternatively, if the quantization parameter of a block is greater or less than a certain threshold, the block may be used for retraining.

As another example, the video encoding device may signal a sample or video signal area to which retraining is applied to the video decoding device.

As another example, a separate trained discriminator may be used. The video block sampling unit 1120 may input the decoded block into an identifier and determine whether to use the block for retraining based on the output of the identifier.

Meanwhile, the video encoding device can control retraining in the video decoding device using a flag indicating whether to retrain (hereinafter referred to as 'retraining flag'). Depending on whether coding efficiency can be improved by updating the deep learning module using video samples available on the decoder side, the video encoding device can set the value of the retraining flag. At this time, when the retraining flag is 0 on a slice or block basis, the video decoding device does not perform retraining of the deep learning module and determines whether to apply the deep learning module based on the in-loop filter flag. On the other hand, when the retraining flag is 1, the video decoding device can perform retraining of the deep learning module. At this time, the in-loop filter flag must also be 1. That is, when the in-loop filter flag is true and the retraining flag is true, the video decoding device can perform retraining of the deep learning module.

As described above, after updating the parameters of the deep learning module using the original video, instead of transmitting the parameters, the video encoding device determines whether to use the updated parameters on the decoder side. Additionally, the video encoding device may set the value of the retraining flag according to the determination result and then signal the retraining flag to the video decoding device.

Hereinafter, a method by which the retraining unit 1130 retrains a deep learning-based in-loop filter will be described.

As described above, when both the in-loop filter flag and the retraining flag are true, the retraining unit 1130 may retrain the deep learning-based in-loop filter.

As an example, the retraining unit 1130 may update parameters within the deep learning module using the supervised learning method described above. In the example of FIG. 12, the loss function L is defined as shown in Equation 4 based on the difference between the current sample x _rec and the corresponding output x _{rec_f} .

The retraining unit 1130 may calculate a gradient value in the direction of reducing the loss function and then update the parameters within the deep learning module using the above-described SGD algorithm.

As another example, the retraining unit 1130 may replace the corresponding output x _{rec_f} in Equation 4 and use the decoded block as the target block. For example, a block located (colocated) corresponding to the current sample in the reference frame of the current sample may be used as the target block. In the case of inter prediction, the reference block indicated by the motion vector of the current sample in the reference frame can be used as the target block. In the case of intra prediction, a prediction block searched according to template matching in the current frame based on the template of the current sample can be used as a target block. Alternatively, a block generated by a weighted sum of all or part of the above-mentioned output x _{rec_f} , the corresponding collocated block, and the reference block indicated by the motion vector may be used as the target block. Alternatively, a block generated by weighting the above-described output x _{rec_f} and the prediction block searched according to template matching may be used as the target block.

As another example, the retraining unit 1130 may update parameters within the deep learning module using the aforementioned contrast learning. The block selected by the video block sampling unit 1120 is used as a positive sample. Additionally, blocks not selected by the video block sampling unit 1120 may be used as negative samples. A positive pair can then be created by combining the two positive samples. Additionally, a negative pair can be created by combining one positive sample and one negative sample. The retraining unit 1130 may update parameters in the deep learning module using the loss function of Equation 1 based on the generated positive and negative pairs. In Equation 1, for example, f() may be a deep learning-based in-loop filter. Therefore, the similarity of a positive pair can be defined as the similarity between outputs generated by inputting positive samples constituting the positive pair into an in-loop filter. Additionally, the similarity of a negative pair can be defined as the similarity between outputs generated by inputting samples constituting the negative pair into an in-loop filter.

Meanwhile, one deep learning module can be used when retraining an in-loop filter. As shown in the example of FIG. 13, the retraining unit 1130 generates x _{rec_f} by inputting the current sample x _rec into an existing deep learning module. Thereafter, the retraining unit 1130 may use the input and/or output of the deep learning module for retraining based on the learning method. The retraining unit 1130 applies the parameters updated during the retraining process to the existing deep learning module. In the example of Figure 13, output x _{rec_s} represents the output of the updated deep learning module.

As another example, two deep learning modules can be used when retraining an in-loop filter. Hereinafter, the two deep learning modules are referred to as a first deep learning module and a second deep learning module. The first deep learning module represents an existing deep learning module. In the initial state, the first deep learning module and the second deep learning module include the same parameters.

As shown in the example of FIG. 14, the retraining unit 1130 generates x _{rec_f} by inputting the current sample x _rec to the first deep learning module. Thereafter, the retraining unit 1130 may use the input and/or output of the deep learning module for retraining based on the learning method. The retraining unit 1130 applies the parameters updated during the retraining process to the second deep learning module. In the example of Figure 14, output x _{rec_s} represents the updated output of the second deep learning module. In the example of FIG. 14, in addition to the retrained parameters, the parameters of the previously used deep learning module may be preserved. In the example of FIG. 13, the current deep learning module is used after being updated, and in the example of FIG. 14, both the first deep learning module and the second deep learning module can be used.

The retraining unit 1130 can use the updated parameters as is. Alternatively, the retraining unit 1130 may perform a weighted average of the existing parameters and the updated parameters and then use the weighted average parameters.

Samples used for retraining of the deep learning module can be stored in video sequence units and then used for retraining. Alternatively, samples used for retraining may be stored in picture, slice, or block units and then used for retraining.

As described above, model-based in-loop filters such as ALF (566), SAO (564), etc. are used in VVC. At this time, the video encoding device derives model parameters using the original video and then transmits the derived parameters to the video decoding device. Hereinafter, using the ALF 566 as an example, a method for an image decoding device to derive model parameters using a restored image will be described.

For the above-described ALF, the video encoding device can derive the ith filter coefficient c _i according to Equation 5.

Here, N represents the number of filter coefficients. x is the pixel value of the original image and y _i is the pixel value of the reconstructed image before being input to the ALF corresponding to the filter coefficient c _i .

The video decoding device can generate updated filter coefficients c _i by using the pixel value x ^j corresponding to the output of the existing ALF instead of x in Equation 5.

For example, when the aforementioned ALF flag is true, the ALF in the video decoding device generates improved output from the pixel values of the reconstructed video. At this time, the video decoding device can update the ALF filter coefficients by using the pixel value x ^j corresponding to the output of the existing ALF for the next input video signals. Alternatively, the improved ALF can be used even for the current sample.

Meanwhile, the video encoding device can control the update of filter coefficients in the video decoding device using a flag indicating whether to update the ALF (hereinafter referred to as 'ALF update flag'). Depending on whether coding efficiency can be improved by updating filter coefficients using the ALF output available on the decoder side, the video encoding device can set the value of the ALF update flag. At this time, when the ALF update flag is 0 on a slice or block basis, the video decoding device does not update filter coefficients and determines whether to apply ALF based on the flag indicating whether to apply ALF. On the other hand, when the ALF update flag is 1, the video decoding device can update filter coefficients. At this time, the flag indicating whether to apply ALF must also be 1. That is, when the flag indicating whether to apply ALF is true and the ALF update flag is true, the video decoding device can update the filter coefficient.

As described above, instead of deriving the filter coefficients of the ALF using the original video and then transmitting the corresponding coefficients, the video encoding device determines whether to use the filter coefficients updated by the decoder. Additionally, the video encoding device may set the value of the ALF update flag according to the determination result and then signal the ALF update flag to the video decoding device.

Hereinafter, using the illustrations of FIGS. 15 and 16, a method of retraining a deep learning-based in-loop filter on the decoder side by a video encoding device or a video decoding device will be described.

FIG. 15 is a flowchart showing a method of encoding a current block performed by a video encoding device according to an embodiment of the present disclosure.

The video encoding device generates a restored block for the current block (S1500).

The image encoding device generates a first output block by inputting the restored block into a deep learning-based in-loop filter (S1502). Here, the in-loop filter is trained in advance to generate an output that is close to the original block from the restored block.

The video encoding device determines the in-loop filter flag based on the reconstruction block and the first output block (S1504).

The video encoding device can determine the value of the in-loop filter flag in terms of bit rate distortion optimization. For example, when the reconstruction block is optimal, the video encoding device sets the in-loop filter flag to false. On the other hand, when the first output block is optimal, the video encoding device can set the in-loop filter flag to true.

The video encoding device encodes the in-loop filter flag (S1506).

The video encoding device checks the in-loop filter flag (S1508).

If the in-loop filter flag is true (Yes in S1508), the video encoding device performs the following steps (S1510 to S1518). On the other hand, if the in-loop filter flag is false (No in S1508), the video encoding device can omit the steps of retraining the in-loop filter.

The video encoding device selects a block for retraining the in-loop filter from the restoration block (S1510).

The video encoding device retrains the in-loop filter using the selected reconstruction block and the corresponding first output block (S1512).

When performing retraining based on the loss function shown in Equation 4, the video encoding device can replace the corresponding output block and use the decoded block as the target block. For example, a block at a position corresponding to the current sample in the reference frame of the current sample may be used as the target block. In the case of inter prediction, the reference block indicated by the motion vector of the current sample in the reference frame can be used as the target block. Alternatively, in the case of intra prediction, a prediction block searched according to template matching in the current frame based on the template of the current sample may be used as the target block.

The video encoding device generates a second output block by inputting the selected restored block into the retrained in-loop filter (S1514).

Alternatively, the video encoding device may apply the retrained in-loop filter to the reconstructed block selected after the current block. That is, the image encoding device can generate a first output block and a second output block for the restored block selected after the current block.

The video encoding device determines a retraining flag based on the first output block and the second output block corresponding to the selected reconstruction block (S1516).

The video encoding device can determine the value of the retraining flag in terms of bit rate distortion optimization. For example, if the first output block is optimal, the video encoding device sets the retraining flag to false. On the other hand, if the second output block is optimal, the video encoding device can set the retraining flag to true.

The video encoding device encodes the retraining flag (S1518).

FIG. 16 is a flowchart showing a method for restoring a current block performed by an image decoding device according to an embodiment of the present disclosure.

The video decoding device generates a restored block for the current block (S1600).

The video decoding device decodes the in-loop filter flag from the bitstream (S1602).

The video decoding device checks the in-loop filter flag (S1604).

If the in-loop filter flag is true (Yes in S1604), the video decoding device can use a deep learning-based in-loop filter. On the other hand, if the in-loop filter flag is false (No in S1604), the video decoding device can omit the use of the in-loop filter.

The video decoding device generates an output block by inputting the restoration block for the current block into a deep learning-based in-loop filter (S1606). Here, the in-loop filter is trained in advance to generate an output that is close to the original block from the restored block.

The video decoding device decodes the retraining flag from the bitstream (S1608).

The video decoding device checks the retraining flag (S1610).

If the retraining flag is true (Yes in S1610), the video decoding device may perform steps (S1612 to S1616) of retraining the in-loop filter. On the other hand, if the retraining flag is false (No in S1610), the video decoding device can omit the steps of retraining the in-loop filter.

The video decoding device selects a block for retraining the in-loop filter from the restoration block (S1612).

The video decoding device retrains the in-loop filter using the selected restoration block and the corresponding output block (S1614).

When performing retraining based on the loss function shown in Equation 4, the video decoding device can replace the corresponding output block and use the decoded block as the target block. For example, a block at a position corresponding to the current sample in the reference frame of the current sample may be used as the target block. In the case of inter prediction, the reference block indicated by the motion vector of the current sample in the reference frame can be used as the target block. Alternatively, in the case of intra prediction, a prediction block searched according to template matching in the current frame based on the template of the current sample may be used as the target block.

The video decoding device generates an improved output block by inputting the selected restored block into the retrained in-loop filter (S1616).

Alternatively, the video decoding device may apply the retrained in-loop filter to the restored block selected after the current block. That is, the image decoding device can generate an improved output block for the restored block selected after the current block.

In the flowchart/timing diagram of this specification, each process is described as being executed sequentially, but this is merely an illustrative explanation of the technical idea of an embodiment of the present disclosure. In other words, a person skilled in the art to which an embodiment of the present disclosure pertains may change the order described in the flowchart/timing diagram and execute one of the processes without departing from the essential characteristics of the embodiment of the present disclosure. Since the above processes can be modified and modified in various ways by executing them in parallel, the flowchart/timing diagram is not limited to a time series order.

It should be understood from the above description that the example embodiments may be implemented in many different ways. The functions or methods described in one or more examples may be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "...units" to particularly emphasize their implementation independence.

Meanwhile, various functions or methods described in this embodiment may be implemented with instructions stored in a non-transitory recording medium that can be read and executed by one or more processors. Non-transitory recording media include, for example, all types of recording devices that store data in a form readable by a computer system. For example, non-transitory recording media include storage media such as erasable programmable read only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid state drives (SSD).

The above description is merely an illustrative explanation of the technical idea of the present embodiment, and those skilled in the art will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these examples. The scope of protection of this embodiment should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this embodiment.

510: Entropy decoding unit
540: prediction unit
560: Loop filter unit
566:ALF
1110: Video block decoding unit
1120: Video block sampling unit
1130: Retraining department

Claims (16)

In the method of restoring the current block performed by the video decoding device,
generating a restored block for the current block from a bitstream;
Inputting the restored block into a deep learning-based in-loop filter to generate an output block, wherein the in-loop filter is pre-trained to generate an output close to the original block from the restored block;
selecting a block for retraining the in-loop filter from the restoration block; and
Retraining the in-loop filter using the selected restoration block and target block.
A method comprising: According to paragraph 1,
decoding an in-loop filter flag from the bitstream; and
Checking the in-loop filter flag
Including more,
When the in-loop filter flag is true, the step of generating the output block is performed. According to paragraph 1,
decoding a retraining flag from the bitstream; and
Checking the retraining flag
Including more,
When the retraining flag is true, selecting a block for retraining and retraining the in-loop filter. According to paragraph 3,
When the retraining flag is true, the method further comprises inputting the selected restoration block into the updated in-loop filter to generate an improved output block. According to paragraph 1,
The step of selecting a block for retraining is:
A method characterized in that it determines whether to select the restored block as a block for the retraining based on parameters used when compressing the restored block. According to paragraph 1,
Further comprising decoding region information to which the retraining is applied from the bitstream,
The step of selecting a block for retraining is:
When the restored block is included in the area information, the method is characterized in that the restored block is selected as a block for the retraining. According to paragraph 1,
The step of selecting a block for retraining is:
A method characterized by inputting the restored block into a pre-trained identifier (discriminator) to generate an output, and then determining whether to select the restored block as a block for retraining based on the output of the identifier. According to paragraph 1,
The target block is,
Based on an output block corresponding to the selected restored block, a block collocated to a position corresponding to the current block in the reference frame of the current block, a reference block indicated by the motion vector of the current block, or a template of the current block. A method characterized in that the prediction block is searched according to template matching in the current frame. According to paragraph 1,
The step of retraining the in-loop filter is,
A method characterized by defining a loss function based on the difference between the selected restoration block and the target block, and updating parameters of the in-loop filter in a direction to reduce the loss function. According to paragraph 1,
The in-loop filter is,
Comprising a first deep learning module and a second deep learning module including the same parameters in an initial state,
The step of generating the output block is,
Generating the output block from the restoration block using the first deep learning module,
The step of retraining the in-loop filter is,
Characterized in that updating parameters of the second deep learning module using the selected restoration block and the target block. In the method of encoding the current block performed by the video encoding device,
generating a restoration block for the current block;
Generating a first output block by inputting the restored block into a deep learning-based in-loop filter, wherein the in-loop filter is trained in advance to generate an output close to the original block from the restored block;
selecting a block for retraining the in-loop filter from the restoration block;
retraining the in-loop filter using the selected restoration block and target block; and
Generating a second output block by inputting the selected restored block into the retrained in-loop filter.
A method comprising: According to clause 11,
determining an in-loop filter flag based on the restoration block and the first output block; and
Encoding the in-loop filter flag
A method further comprising: According to clause 12,
Further comprising checking the in-loop filter flag,
When the in-loop filter flag is true, the method includes selecting a block for retraining, retraining the in-loop filter, and generating the second output block. According to clause 12,
determining a retraining flag based on a first output block corresponding to the selected restoration block and the second output block; and
Encoding the retraining flag
A method further comprising: According to clause 11,
The target block is,
A first output block corresponding to the selected restored block, a block located (collocated) corresponding to the current block in the reference frame of the current block, a reference block indicated by the motion vector of the current block, or a template of the current block. A method characterized in that the prediction block is searched according to template matching in the current frame based on . A computer-readable recording medium storing a bitstream generated by an image encoding method, the image encoding method comprising:
Generating a restoration block for the current block;
Generating a first output block by inputting the restored block into a deep learning-based in-loop filter, wherein the in-loop filter is trained in advance to generate an output close to the original block from the restored block;
selecting a block for retraining the in-loop filter from the restoration block;
retraining the in-loop filter using the selected restoration block and target block; and
Generating a second output block by inputting the selected restored block into the retrained in-loop filter.
A recording medium comprising:

Images