KR20240025420A - Image decoding apparatus and image encoding apparatus using ai, and methods thereby - Google Patents

Abstract

A method of decoding an image according to an embodiment includes obtaining a motion vector of a current block; Obtaining a preliminary prediction block using a reference block indicated by a motion vector in a reference image; Obtaining a final prediction block of the current block by applying at least one of a POC map, a preliminary prediction block, or a quantization error map including a POC difference between a current image including the current block and a reference image to a neural network; and restoring the current block using the residual block and the final prediction block obtained from the bitstream, where sample values of the quantization error map may be calculated from the quantization parameter for the reference block.

Description

Video decoding device, video encoding device and method using AI using AI {IMAGE DECODING APPARATUS AND IMAGE ENCODING APPARATUS USING AI, AND METHODS THEREBY}

The present disclosure relates to a method and device for processing an image, and more specifically, to a method and device for encoding/decoding an image using AI (Artificial Intelligence).

In codecs such as H.264 AVC (Advanced Video Coding) and HEVC (High Efficiency Video Coding), the video is divided into blocks, and the divided blocks are predicted and encoded through inter prediction or intra prediction. and predictive decoding.

Intra prediction is a method of compressing images by removing spatial redundancy within the images, and inter prediction is a method of compressing images by removing temporal redundancy between images.

In the encoding process, a prediction block of the current block is generated through intra-prediction or inter-prediction, a residual block is generated by subtracting the prediction block from the current block, and residual samples of the residual block are transformed and quantized.

In the decoding process, the quantized transform coefficients of the residual block are inversely quantized and inversely transformed to generate residual samples of the residual block, and the prediction block generated through intra prediction or inter prediction is added to the residual block to restore the current block. The restored current block may be processed according to one or more filtering algorithms and then output.

Codecs such as H.264 AVC and HEVC use a rule-based prediction mode for inter prediction of the current block. Rule-based prediction modes may include, for example, skip mode, merge mode, and advanced motion vector prediction (AMVP) mode.

Traditionally, rule-based prediction modes have shown good performance, but as the resolution of images increases and the content of images becomes more diverse, an AI-based prediction mode that can flexibly consider the characteristics of the images may be required.

A method of decoding an image according to an embodiment may include obtaining a motion vector of a current block.

In one embodiment, a method of decoding an image may include obtaining a preliminary prediction block using a reference block indicated by a motion vector in a reference image.

In one embodiment, a method of decoding an image includes sending at least one of a POC (Picture Order Difference) map including a POC (Picture Order Difference) difference between a current image including a current block, a preliminary prediction block, or a quantization error map to a neural network. It may include a step of applying the method to obtain the final prediction block of the current block.

In one embodiment, a method of decoding an image may include restoring a current block using a residual block and a final prediction block obtained from a bitstream.

In one embodiment, sample values of the quantization error map may be calculated from the quantization parameters for the reference block.

A method of encoding an image according to an embodiment may include obtaining a motion vector pointing to a reference block in a reference image corresponding to a current block.

In one embodiment, a method of encoding an image includes at least one of a POC map including a POC difference between a current image including a current block and a reference image, a preliminary prediction block obtained based on the reference block, or a quantization error map. It may include the step of applying to a neural network to obtain the final prediction block of the current block.

In one embodiment, a method of encoding an image may include obtaining a residual block using a current block and a final prediction block.

In one embodiment, a method of encoding an image may include generating a bitstream including information about information about a residual block.

In one embodiment, sample values of the quantization error map may be calculated from the quantization parameters for the reference block.

An image decoding device according to an embodiment includes at least one memory storing at least one instruction; and at least one processor operating according to at least one instruction.

In one embodiment, at least one processor of the video decoding apparatus may obtain a motion vector of the current block.

In one embodiment, at least one processor of the image decoding apparatus may obtain a preliminary prediction block using a reference block indicated by a motion vector in a reference image.

In one embodiment, at least one processor of the image decoding device applies at least one of a POC map, a preliminary prediction block, or a quantization error map including a POC difference between a current image including a current block and a reference image to a neural network to The final prediction block of the current block can be obtained.

In one embodiment, at least one processor of the image decoding apparatus may restore the current block using a residual block and a final prediction block obtained from a bitstream.

In one embodiment, sample values of the quantization error map may be calculated from the quantization parameters for the reference block.

An image encoding device according to an embodiment includes at least one memory storing at least one instruction; and at least one processor operating according to at least one instruction.

In one embodiment, at least one processor of the video encoding device may obtain a motion vector indicating a reference block in a reference image corresponding to the current block.

In one embodiment, at least one processor of the image encoding device may select one of a POC map including a POC difference between a current image including a current block and a reference image, a preliminary prediction block obtained based on a reference block, or a quantization error map. At least one can be applied to the neural network to obtain the final prediction block of the current block.

In one embodiment, at least one processor of the image encoding device may obtain a residual block using the current block and the final prediction block.

In one embodiment, at least one processor of an image encoding device may generate a bitstream including information about information about a residual block.

In one embodiment, sample values of the quantization error map may be calculated from the quantization parameters for the reference block.

FIG. 1 is a diagram illustrating the configuration of a video decoding device according to an embodiment.
Figure 2 is a diagram illustrating the configuration of an AI-based prediction decoder according to an embodiment.
FIG. 3 is a diagram illustrating spatial neighboring blocks and temporal neighboring blocks of the current block according to an embodiment.
FIG. 4 is a diagram illustrating reference blocks indicated by a motion vector for list 0 and a motion vector for list 1 according to an embodiment.
FIG. 5 is a diagram illustrating a method of obtaining a quantization error map based on quantization parameters for a reference block according to an embodiment.
FIG. 6 is a diagram illustrating a method of obtaining a quantization error map based on quantization parameters for a reference block according to an embodiment.
FIG. 7 is a diagram illustrating a method of obtaining a quantization error map based on quantization parameters for a reference block according to an embodiment.
FIG. 8 is a diagram illustrating a method of obtaining a quantization error map based on quantization parameters for a reference block according to an embodiment.
Figure 9 is a diagram showing the structure of a neural network according to one embodiment.
Figure 10 is a diagram for explaining a convolution operation in a convolution layer according to an embodiment.
FIG. 11 is a diagram illustrating a method of obtaining an extended preliminary prediction block according to an embodiment.
FIG. 12 is a diagram illustrating a method of obtaining an extended preliminary prediction block when the boundary of a reference block corresponds to the boundary of a reference image, according to an embodiment.
Figure 13 is a diagram for explaining a method of obtaining an extended quantization error map according to an embodiment.
FIG. 14 is a diagram illustrating reconstructed and unrestored samples in a current video according to an embodiment.
FIG. 15 is a diagram illustrating a method of obtaining an extended current restored block according to an embodiment.
Figure 16 is a diagram illustrating a plurality of weight sets according to one embodiment.
FIG. 17 is a diagram illustrating an image decoding method using an image decoding device according to an embodiment.
Figure 18 is a diagram illustrating syntax according to one embodiment.
FIG. 19 is a diagram illustrating the configuration of a video encoding device according to an embodiment.
FIG. 20 is a diagram illustrating the configuration of an AI-based prediction encoder according to an embodiment.
FIG. 21 is a diagram illustrating a method of changing a fractional precision motion vector to an integer precision motion vector according to an embodiment.
FIG. 22 is a diagram illustrating an image encoding method using an image encoding device according to an embodiment.
Figure 23 is a diagram for explaining a neural network training method according to an embodiment.
Figure 24 is a diagram for explaining the overall encoding and decoding process of an image according to an embodiment.

Since the present disclosure can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail through detailed description. However, this is not intended to limit the embodiments of the present disclosure, and the present disclosure may include all changes, equivalents, and substitutes included in the spirit and technical scope of the various embodiments.

In describing the embodiments, if it is determined that detailed descriptions of related known technologies may unnecessarily obscure the gist of the present disclosure, the detailed descriptions may be omitted. In addition, numbers (eg, first, second, etc.) used in the description of the specification correspond to identification symbols for distinguishing one component from another component.

In the present disclosure, the expression “at least one of a, b, or c” refers to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b and c", or variations thereof.

In the present disclosure, when a component is referred to as “connected” or “connected” to another component, the component may be directly connected or directly connected to the other component, but in particular, the contrary is not specified. Unless a base material exists, it may be connected or connected through another component in the middle.

In the present disclosure, components expressed as '~unit (unit)', 'module', etc. are two or more components combined into one component, or one component divided into two or more for more detailed functions. It could be. Each of the components described below may additionally perform some or all of the functions performed by other components in addition to the main functions that each component is responsible for, and some of the main functions performed by each component may be performed by other components. It may also be carried out in full charge by .

In the present disclosure, ‘image or picture’ may mean a still image (or frame), a moving image composed of a plurality of consecutive still images, or a video.

In this disclosure, 'neural network' is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model using a specific algorithm. A neural network may also be referred to as a deep neural network.

In the present disclosure, 'weight' is a value used in the calculation process of each layer forming a neural network, and can be used, for example, when applying an input value to a predetermined calculation equation. The weight is a value set as a result of training, and can be updated through separate training data as needed.

In this disclosure, 'current block' refers to a block that is currently being processed. The current block may be a slice, tile, maximum coding unit, coding unit, prediction unit, or transformation unit divided from the current image.

In the present disclosure, 'sample' refers to data assigned to a sampling location in data such as an image, block, filter kernel, or map, and is the data to be processed. For example, a sample may include pixels in a two-dimensional image.

Before describing the video decoding device 100 and the video encoding device 1900 according to an embodiment, the overall video encoding and decoding process will be described with reference to FIG. 24.

FIG. 24 is a diagram illustrating the overall encoding and decoding process of an image according to an embodiment.

The encoding device 2410 transmits a bitstream generated by encoding an image to the decoding device 2450, and the decoding device 2450 can restore the image by receiving and decoding the bitstream.

In one embodiment, the prediction encoder 2415 of the encoding device 2410 outputs a prediction block through inter-prediction or intra-prediction for the current block, and the transform and quantization unit 2420 outputs a prediction block between the prediction block and the current block. The residual samples of the residual block can be transformed and quantized to output quantized transform coefficients.

The entropy encoder 2425 can encode the quantized transform coefficient and output it as a bitstream.

The quantized transform coefficient can be restored into a residual block including residual samples in the spatial domain through the inverse quantization and inverse transform unit 2430. The restored block, which is a combination of the prediction block and the residual block, may be output as a filtered block through the deblocking filtering unit 2435 and the loop filtering unit 2440. The reconstructed image including the filtered block can be used as a reference image for the next image in the prediction encoder 2415.

The bitstream received by the decoding device 2450 can be restored into a residual block including residual samples in the spatial domain through the entropy decoding unit 2455 and the inverse quantization and inverse transform unit 2460.

The prediction block and the residual block output from the prediction decoder 2475 are combined to generate a restored block, and the restored block can be output as a filtered block through the deblocking filtering unit 2465 and the loop filtering unit 2470. . The reconstructed image including the filtered block can be used by the prediction decoder 2475 as a reference image for the next image.

In one embodiment, the prediction encoder 2415 and the prediction decoder 2475 may predictively encode and decode the current block according to a rule-based prediction mode and/or a neural network-based prediction mode.

In one embodiment, the rule-based prediction mode includes merge mode, skip mode, advanced motion vector prediction (AMVP) mode, bi-directional optical flow (BDOF) mode, or bi-prediction with CU-level weights (BCW), etc. can do.

In one embodiment, the prediction encoder 2415 and the prediction decoder 2475 may apply a rule-based prediction mode and a neural network-based prediction mode to the current block.

A neural network-based prediction mode according to an embodiment will be described in detail with reference to FIGS. 1 to 23.

Referring to FIG. 1, the video decoding device 100 may include a bitstream parsing unit 110 and a decoding unit 130. The decoding unit 130 may include an AI-based prediction decoding unit 132 and a restoration unit 134.

In one embodiment, the bitstream parsing unit 110 may correspond to the entropy decoding unit 2455 shown in FIG. 24. In one embodiment, the decoding unit 130 may correspond to the inverse quantization and inverse transform unit 2460, prediction decoding unit 2475, deblocking filtering unit 2465, and loop filtering unit 2470 shown in FIG. 24. .

The bitstream parsing unit 110 and the decoding unit 130 may be implemented with at least one processor. The bitstream parsing unit 110 and the decoding unit 130 may operate according to at least one instruction stored in at least one memory.

Although FIG. 1 shows the bitstream parsing unit 110 and the decoding unit 130 separately, the bitstream parsing unit 110 and the decoding unit 130 may be implemented through one processor. In this case, the bitstream parsing unit 110 and the decoding unit 130 may be implemented with a dedicated processor, or a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphic processing unit (GPU) and software. It can also be implemented through a combination of . Additionally, in the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure, or a memory processing unit for using an external memory.

In one embodiment, the bitstream parsing unit 110 and the decoding unit 130 may be comprised of a plurality of processors. In this case, it may be implemented through a combination of dedicated processors, or it may be implemented through a combination of software and multiple general-purpose processors such as AP, CPU, or GPU.

The bitstream parsing unit 110 may obtain a bitstream including an encoding result for an image.

The bitstream parsing unit 110 may receive a bitstream from the video encoding device 1900 through a network.

In one embodiment, the bitstream parsing unit 110 may be used in magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. A bitstream can also be obtained from a data storage medium including a (magneto-optical medium).

The bitstream parsing unit 110 may parse the bitstream to obtain information necessary to restore the image.

In one embodiment, the bitstream parsing unit 110 may obtain syntax elements for image reconstruction from the bitstream. Binary values corresponding to syntax elements may be included in the bitstream according to the hierarchical structure of the image. The bitstream parsing unit 110 may obtain syntax elements by entropy coding binary values included in the bitstream.

In one embodiment, the bitstream parsing unit 110 may transmit information about the motion vector and information about the residual block obtained from the bitstream to the decoding unit 130.

The decoder 130 may obtain the current restored block based on the information delivered from the bitstream parsing unit 110. The current restored block may refer to a block obtained through decoding the encoded current block.

In one embodiment, the AI-based prediction decoder 132 may obtain the final prediction block of the current block using information about the motion vector.

The AI-based prediction decoder 132 may obtain the final prediction block of the current block using AI, for example, a neural network.

The mode in which the AI-based prediction decoder 132 obtains the final prediction block of the current block using a neural network can be defined as a neural network-based prediction mode.

The restoration unit 134 may obtain the residual block of the current block using information about the residual block provided from the bitstream parsing unit 110. In one embodiment, information about the residual block may include information about quantized transform coefficients.

In one embodiment, the restoration unit 134 may obtain a residual block in the spatial domain by inversely quantizing and inversely transforming the quantized transform coefficient.

The reconstruction unit 134 may obtain a current reconstruction block corresponding to the current block using the final prediction block and the residual block. In one embodiment, the reconstruction unit 134 may obtain the current reconstruction block by adding the sample values of the final prediction block and the sample values of the residual block.

Below, with reference to FIG. 2, the AI-based prediction decoder 132 will be described in more detail.

FIG. 2 is a diagram illustrating the configuration of the AI-based prediction decoder 132 according to an embodiment.

Referring to FIG. 2, the AI-based prediction decoder 132 may include a motion information acquisition unit 210, a prediction block acquisition unit 220, a neural network setup unit 230, and a neural network 240.

Neural network 240 may be stored in memory. In one embodiment, neural network 240 may be implemented with an AI processor.

The motion information acquisition unit 210 may obtain the motion vector of the current block using information about the motion vector. As described later, when the precision of the motion vector of the current block is changed from fractional precision to integer precision by the motion information acquisition unit 2010 of the image encoding device 1900, the motion information acquisition unit 210 determines the integer precision of the current block. A motion vector with high precision can be obtained.

Information about the motion vector may include information indicating one or more motion vector candidates among the motion vector candidates included in the motion vector candidate list, for example, a flag or an index. In one embodiment, the information about the motion vector may further include information about a residual motion vector corresponding to the difference between the predicted motion vector of the current block and the motion vector of the current block.

The prediction block acquisition unit 220 may obtain a preliminary prediction block using the motion vector and reference image of the current block acquired by the motion information acquisition unit 210.

In one embodiment, the motion information acquisition unit 210 and the prediction block acquisition unit 220 may obtain a preliminary prediction block of the current block based on a rule-based prediction mode.

Rule-based prediction modes may include merge mode, skip mode, or AMVP mode.

According to the merge mode or skip mode, the motion information acquisition unit 210 constructs a motion vector candidate list including motion vectors of neighboring blocks of the current block as motion vector candidates, and selects a motion vector candidate list among the motion vector candidates included in the motion vector candidate list. The motion vector candidate indicated by the information included in the bitstream can be determined as the motion vector of the current block.

In addition, according to the AMVP mode, the motion information acquisition unit 210 constructs a motion vector candidate list including motion vectors of neighboring blocks of the current block as motion vector candidates, and selects bits from among the motion vector candidates included in the motion vector candidate list. The motion vector candidate indicated by the information included in the stream can be determined as the predicted motion vector of the current block. Additionally, the motion information acquisition unit 210 may determine the motion vector of the current block using the predicted motion vector and the residual motion vector of the current block.

Merge mode, skip mode, or AMVP mode are examples of rule-based prediction modes, and in one embodiment, the rule-based prediction mode may further include decoder-side motion vector refinement (DMVR) mode, etc.

A process of constructing a motion vector candidate list is commonly performed in merge mode, skip mode, and AMVP mode, and neighboring blocks that can be included in the motion vector candidate list will be described with reference to FIG. 3.

Referring to FIG. 3, the neighboring blocks of the current block 300 are spatial neighboring blocks (A0, A1, B0, B1, B2) spatially adjacent to the current block 300, and temporal neighboring blocks temporally adjacent to the current block 300. May include blocks (Col, Br).

In one embodiment, the spatial surrounding block is at least one of a lower left corner block (A0), a lower left block (A1), an upper right corner block (B0), an upper right block (B1), or an upper left corner block (B2). may include.

In one embodiment, the temporal neighboring block is a block ( It may include at least one of Col) or a block (Br) that is spatially adjacent to a block (Col) located at the same point.

The block Br may be located in the lower right corner of the block Col located at the same point as the current block 300. The block Col located at the same point as the current block 300 may be a block that includes a pixel corresponding to the center pixel in the current block 300 among pixels included in the collocated image.

The motion information acquisition unit 210 may determine the availability of neighboring blocks in a predetermined order, and sequentially include the motion vectors of the neighboring blocks as motion vector candidates in the motion vector candidate list according to the determination result.

In one embodiment, the motion information acquisition unit 210 may determine that the block is not available if the neighboring block is intra-predicted.

In one embodiment, the motion vector acquired by the motion information acquisition unit 210 may include a motion vector for list 0, a motion vector for list 1, or a motion vector for list 0 and a motion vector for list 1. there is.

The motion vector for list 0 is a motion vector pointing to a reference block in the reference image included in list 0 (or reference image list 0), and the motion vector for list 1 is included in list 1 (or reference image list 1). It may be a motion vector to indicate a reference block in the reference image.

The prediction block acquisition unit 220 may obtain a preliminary prediction block using the reference block indicated by the motion vector in the reference image.

In one embodiment, the preliminary prediction block may be obtained by applying interpolation to the reference block indicated by the motion vector within the reference image. Accordingly, the preliminary prediction block may include subpixels obtained by applying filtering to integer pixels.

In one embodiment, the reference block indicated by the motion vector within the reference image may be determined as a preliminary prediction block. For example, if the precision of the motion vector of the current block is integer precision, the reference block indicated by the motion vector may be determined as the preliminary prediction block.

When the motion vector for list 0 is acquired by the motion information acquisition unit 210, the prediction block acquisition unit 220 acquires the reference block indicated by the motion vector for list 0 within the reference image included in list 0, , a preliminary prediction block for list 0 can be obtained using the corresponding reference block.

In one embodiment, when a motion vector for List 1 is acquired by the motion information acquisition unit 210, the prediction block acquisition unit 220 determines the motion vector indicated by the motion vector for List 1 within the reference image included in List 1. A reference block can be obtained, and a preliminary prediction block for List 1 can be obtained using the reference block.

In one embodiment, when a motion vector for list 0 and a motion vector for list 1 are acquired by the motion information acquisition unit 210, the prediction block acquisition unit 220 obtains the list within the reference image included in list 0. Obtain a prediction block for list 0 using the reference block pointed to by the motion vector for list 0, and obtain a prediction block for list 1 using the reference block pointed to by the motion vector for list 1 within the reference image included in list 1. can be obtained.

FIG. 4 is a diagram illustrating reference blocks indicated by a motion vector for list 0 and a motion vector for list 1 according to an embodiment.

As shown in FIG. 4, when a motion vector (mv1) for list 0 and a motion vector (mv2) for list 1 are obtained for the current block 300 in the current image 400, the prediction block acquisition unit ( 220) acquires the first reference block 415 indicated by the motion vector (mv1) for list 0 within the first reference image 410 included in list 0, and the second reference image 430 included in list 1 ), the second reference block 435 indicated by the motion vector (mv2) for list 1 can be obtained. A first preliminary prediction block for list 0 may be obtained from the first reference block 415, and a second preliminary prediction block for list 1 may be obtained from the second reference block 435.

Referring again to FIG. 2, the neural network setting unit 230 may obtain data to be input to the neural network 240.

In one embodiment, the neural network setting unit 230 may obtain data to be input to the neural network 240 based on quantization parameters for the reference image, the preliminary prediction block, and the reference block.

In one embodiment, at least one of a preliminary prediction block, a POC map, or a quantization error map may be input to the neural network 240 by the neural network setting unit 230.

Describing the data input to the neural network 240 in detail, the preliminary prediction block is a block determined to be similar to the current block under a rule-based prediction mode, and can be used to obtain a final prediction block that is more similar to the current block.

The quantization parameter for the reference block can be used to quantize/dequantize the residual data of the reference block in the process of encoding/decoding the reference block. Depending on the quantization parameter, the amount of error due to quantization/dequantization may vary. In other words, the quantization parameter can be said to mean the amount of error or distortion included in the reference block restored through encoding/decoding.

By inputting the value calculated based on the quantization parameter to the neural network 240, the neural network 240 determines the reliability of the samples of the preliminary prediction block or the reliability of the samples of the preliminary prediction block in obtaining the final prediction block from the preliminary prediction block. The influence on the samples can be considered. As will be described later, the neural network 240 operates in a direction where the difference between the final prediction block for training, which is output based on the quantization error map for training and the preliminary prediction block for training, and the current block (or original block) for training becomes smaller. It can be trained. Accordingly, the neural network 240 may check the effect of the quantization error map for training on the final prediction block for training and output a final prediction block for training that is similar to the current block for training.

The POC map may include the difference between the picture order count (POC) of the current image including the current block and the POC of the reference image (hereinafter referred to as POC difference) as sample values. POC can indicate the output order of images. Therefore, the POC difference between the current image and the reference image may mean a difference in the output order or a temporal difference between the current image and the reference image. Because the movement of an object may cause a change in the position or size of the object in consecutive images, the neural network 240 learns the influence of the temporal difference between the current image and the reference image to create a final prediction block that is more similar to the current block. Can be printed.

In one embodiment, neural network 240 may include one or more convolutional layers. The neural network 240 may process at least one of a preliminary prediction block, a POC map, or a quantization error map input from the neural network setting unit 230 and output a final prediction block.

As will be described later with reference to FIGS. 9 and 10 , in the neural network 240, sample values of the final prediction block may be individually determined by applying a predetermined operation to the input data. In the rule-based prediction mode, a motion vector is calculated for each block of the image, whereas in the neural network-based prediction mode according to one embodiment, the samples of the final prediction block are determined individually, which means that the neural network 240 determines the motion for the current block. It can be understood as considering vectors on a sample-by-sample basis. Therefore, according to the neural network-based prediction mode according to one embodiment, a final prediction block that is more similar to the current block can be obtained compared to the rule-based prediction mode.

Hereinafter, a method of obtaining a quantization error map input to the neural network 240 will be described with reference to FIGS. 5 to 8.

5 to 8 are diagrams for explaining a method of obtaining a quantization error map based on quantization parameters for a reference block according to an embodiment.

In one embodiment, sample values of the quantization error map may be calculated from the quantization parameters for the reference block.

Quantization parameters for a reference block can be obtained from a bitstream containing information for decoding the reference block.

In one embodiment, the quantization error map may include quantization error values calculated from quantization parameters as sample values.

Quantization error values may represent the amount of error that may be caused by quantization and dequantization applied to residual samples during the encoding and decoding process for the reference block.

A large quantization error value may mean that the difference between the transform coefficient before quantization and the transform coefficient after dequantization may be large. As the difference between the transform coefficient before quantization and the transform coefficient after dequantization is larger, the identity between the original block and the reference block obtained through decoding the encoded data may be reduced.

Since errors caused by quantization and dequantization are artifacts, neural network-based inter prediction needs to be performed taking quantization error values into consideration.

In one embodiment, the quantization error value can be calculated from Equation 1 below.

[Equation 1]

Quantization error value = Quantization step size^2 / 12

Referring to Equation 1, the quantization error value may be proportional to the square of the quantization step size.

The quantization step size is a value used to quantize the transform coefficient, and the transform coefficient can be quantized by dividing the transform coefficient by the quantization step size. Conversely, the quantized transform coefficient can be dequantized by multiplying the quantized transform coefficient by the quantization step size.

The quantization step size can be approximated by Equation 2 below.

[Equation 2]

Quantization step size = 2^(quantization parameter/n) / quantization scale[quantization parameter%n]

In Equation 2, quantization scale [quantization parameter %n] represents the scale value indicated by the quantization parameter among n predetermined scale values. The HEVC codec defines six scale values (26214, 23302, 20560, 18396, 16384, and 14564), so n is 6 according to the HEVC codec.

Referring to Equation 1 and Equation 2, as the quantization parameter increases, the quantization step size increases and the quantization error value may increase.

In one embodiment, the quantization error map may include the quantization step size calculated from the quantization parameter as a sample value.

Referring to FIG. 5, if the reference block 510 includes 16 samples and the quantization parameter for the 16 samples is a1, the sample values of the quantization error map 530 may have a2 calculated from a1. .

The quantization parameters for the reference block 510 shown in FIG. 5 may be set for the reference block 510, or may be set to an upper block of the reference block 510, e.g., a block containing the reference block 510. Can be set for slices. In other words, the neural network setting unit 230 may obtain sample values of the quantization error map 530 from the quantization parameter set for the reference block 510 or the quantization parameter set for the slice including the reference block 510. .

If the quantization parameter for the reference block 510 is set for an upper block of the reference block 510, for example, a slice including the reference block 510, the quantization parameter is the same for the blocks included in the slice. can be applied. That is, since quantization error maps of blocks included in the slice can be obtained based on the quantization parameter set for the slice, the number of information included in the bitstream is reduced compared to the case of setting the quantization parameter for each block or sample. You can do it.

Next, referring to FIG. 6, quantization parameters for the reference block 510-1 may be set for each sample of the reference block 510-1.

The neural network setting unit 230 may obtain sample values of the quantization error map 530-1 from the quantization parameters for each sample of the reference block 510-1.

The neural network setting unit 230 calculates the value of the upper left sample 631 of the quantization error map 530-1 as a2 from the quantization parameter a1 of the upper left sample 611 of the reference block 510-1, and refers to From the quantization parameter b1 of the sample 612 located to the right of the upper left sample 611 of the block 510-1, the sample 632 located to the right of the upper left sample 631 of the quantization error map 530-1 ) can be calculated as b2.

When quantization parameters are set for each sample of the reference block 510-1, the amount of information that must be obtained from the bitstream to check the quantization parameters for the reference block 510-1 may increase. However, since the final prediction block of the current block is obtained using the reference block 510-1 with few errors, the number of bits for representing the residual block between the current block and the final prediction block can be reduced.

Next, referring to FIG. 7, the neural network setting unit 230 stores the quantization error map 530-2 in sub-regions corresponding to the sub-blocks 710, 720, 730, and 740 of the reference block 510-2. It is divided into fields 750, 760, 770, and 780, and sample values included in each of the sub-regions 750, 760, 770, and 780 of the quantization error map 530-2 are used in the reference block 510-2. It can be calculated from the quantization parameters for samples at predetermined positions within the subblocks 710, 720, 730, and 740.

In one embodiment, if the reference block 510-2 corresponds to a coding unit, the lower blocks 710, 720, 730, and 740 may correspond to a prediction unit.

In one embodiment, the predetermined location may include the upper left location within the sub-block.

As shown in FIG. 7, the sample values of the first sub-region 750 of the quantization error map 530-2 are located at the upper left position among the samples of the first sub-block 710 in the reference block 510-2. It may have a2 calculated from the quantization parameter a1 for the sample 711 of .

Additionally, the sample values of the second sub-region 760 of the quantization error map 530-2 include the sample 721 at the upper left position among the samples of the second sub-block 720 in the reference block 510-2. You can have e2 calculated from the quantization parameter e1.

Additionally, the sample values of the third sub-region 770 of the quantization error map 530-2 include the sample 731 at the upper left position among the samples of the third sub-block 730 in the reference block 510-2. c2 calculated from the quantization parameter c1, and the sample values of the fourth sub-region 780 of the quantization error map 530-2 are the samples of the fourth sub-block 740 in the reference block 510-2. It may have b2 calculated from quantization parameter b1 for the sample 741 at the upper left position.

As an example, a sample at a certain location may include a sample at a central location within a subblock. Here, the sample in the central position means a sample located in the lower left, a sample located in the upper left, a sample located in the lower right, or a sample located in the upper right of the point where the width and height of the predetermined area are divided in half. You can.

FIG. 8 illustrates that the sample at the central location is a sample located at the lower left of the point where the width and height of the predetermined area are divided in half.

As shown in FIG. 8, the sample values of the first sub-region 850 of the quantization error map 530-3 are at the central position among the samples of the first sub-block 710 in the reference block 510-2. It may have a2 calculated from the quantization parameter a1 for the sample 811.

Additionally, the sample values of the second sub-region 860 of the quantization error map 530-3 are for the sample 821 at the center position among the samples of the second sub-block 720 in the reference block 510-2. It may have e2 calculated from the quantization parameter e1.

Additionally, the sample values of the third sub-region 870 of the quantization error map 530-3 are for the sample 831 at the center position among the samples of the third sub-block 730 in the reference block 510-2. It may have a2 calculated from the quantization parameter a1, and the sample values of the fourth sub-region 880 of the quantization error map 830-3 are samples of the fourth sub-block 740 in the reference block 510-2. It may have c2 calculated from the quantization parameter c1 for the sample 841 at the center position.

The sample at the upper left position and the sample at the center position described in relation to FIGS. 7 and 8 are just one example, and in one embodiment, sample values of sub-regions of the quantization error maps 530-2 and 530-3 are The specific location within the subblocks 710, 720, 730, and 740 to be acquired can be changed in various ways.

When the quantization parameters for a reference block are set for each sub-block or for each sample, as in the embodiment described in connection with FIGS. 7 and 8, the sub-blocks 710, 720, 730, and 740 of the reference block 510-2 ) By calculating the sample values of the sub-regions of the quantization error maps 530-2 and 530-3 from the quantization parameters for samples at predetermined positions within the quantization error maps 530-2 and 530-3, the quantization error maps 530-2 and 530-3 can be generated more quickly. You can.

In one embodiment, the neural network setting unit 230 determines the size of the current block, the prediction direction of the current block, the layer to which the current image belongs in the hierarchical structure of the image, or information obtained from the bitstream (e.g., a flag or index). ) Based on at least one of the following, one of the different quantization error map acquisition methods (e.g., the quantization error map acquisition methods shown in FIGS. 5 to 8) is selected, and quantization is performed according to the selected method. An error map can be obtained.

As described above, a final prediction block may be obtained as at least one of a preliminary prediction block, a quantization error map, or a POC map is input to the neural network 240. Referring to FIG. 9, an example structure of the neural network 240 is shown. Explain.

FIG. 9 is a diagram illustrating the structure of a neural network 240 according to one embodiment.

As shown in FIG. 9 , the preliminary prediction block 902, the quantization error map 904, and the POC map 906 may be input to the first convolutional layer 910.

In one embodiment, the size of the preliminary prediction block 902, quantization error map 904, and POC map 906 may be the same as the size of the current block.

Since the POC map 906 includes the POC difference between the current image and the reference image as sample values, all sample values in the POC map 906 may be the same.

The 6 As a result of convolution processing, 32 feature maps can be generated by 32 filter kernels.

The fact that the first convolution layer 910 processes input data of 6 channels takes into account the case where the current block is bidirectionally predicted.

In one embodiment, when the current block is bidirectionally predicted, a first motion vector for list 0 and a second motion vector for list 1 are derived for the current block, and the first motion vector included in list 0 as a reference image of the current block is derived. 1 reference image and a second reference image included in list 1 may be obtained.

In addition, the first preliminary prediction block is obtained from the first reference block in the first reference image indicated by the first motion vector for list 0, and the second reference block in the second reference image indicated by the second motion vector for list 1 A second preliminary prediction block can be obtained from. That is, data input to the neural network 240 includes a first preliminary prediction block, a second preliminary prediction block, a first quantization error map including sample values calculated from quantization parameters for the first reference block, and a second reference block. A second quantization error map including sample values calculated from the quantization parameters for, a first POC map including a POC difference between the current image and the first reference image, and a POC difference between the current image and the second reference image. A second POC map including

The first convolution layer 910 combines the first preliminary prediction block, the second preliminary prediction block, the first quantization error map, the second quantization error map, the first POC map, and the second POC map into 32 filter kernels of 5X5 size. It can be convolutionally processed.

If unidirectional prediction, for example, list 0 direction prediction or list 1 direction prediction, is applied to the current block, in other words, only the first motion vector for list 0 is obtained for the current block or the first motion vector for list 1 is obtained. When only the second motion vector is derived, only three channels of input data can be obtained. Since the number of channels that can be processed in the first convolution layer 910 shown in FIG. 9 is 6, there is a need to increase the input data of 3 channels to input data of 6 channels.

The neural network setting unit 230 copies the first preliminary prediction block (or second preliminary prediction block), the first quantization error map (or second quantization error map), and the first POC map (or second POC map) Two first preliminary prediction blocks (or two second preliminary prediction blocks), two first quantization error maps (or two second quantization error maps) and two first POC maps (or two second POC maps) ) can be obtained, and the input data of 6 channels can be input into the neural network 240.

Feature maps generated by the first convolutional layer 910 may represent unique characteristics of the input data. For example, feature maps may represent vertical characteristics, horizontal characteristics, or edge characteristics of input data.

With reference to FIG. 10 , the convolution operation in the first convolution layer 910 will be described in detail.

Between the weights of the filter kernel 1010 with a size of 5X5 used in the first convolutional layer 910 and the sample values in the corresponding input data 1005 (e.g., preliminary prediction block 902) One feature map 1030 can be created through a multiplication operation and an addition operation.

Since 32 filter kernels are used in the first convolution layer 910, 32 feature maps can be generated through a convolution operation process using 32 filter kernels.

In FIG. 10 , I1 to I49 displayed in the input data 1005 represent samples of the input data 1005, and F1 to F25 displayed in the filter kernel 1010 represent samples of the filter kernel 1010. Additionally, M1 to M9 displayed in the feature map 1030 represent samples of the feature map 1030.

In the convolution operation process, the product of sample values I1 to I5, I8 to I12, I15 to I19, I22 to I26, and I29 to I33 of the input data 1005 and F1 to F25 of the filter kernel 1010 is performed. A value obtained by combining the result values of the multiplication operation (e.g., an addition operation) may be assigned as the value of M1 of the feature map 1030.

If the stride of the convolution operation is 1, each of the sample values I2 to I6, I9 to I13, I16 to I20, I23 to I27, and I30 to I34 of the input data 1005 and F1 to I34 of the filter kernel 1010 Each F25 multiplication operation is performed, and a value obtained by combining the result values of the multiplication operation may be assigned as the value of M2 of the feature map 1030.

A convolution operation is performed between the sample values in the input data 1005 and the samples of the filter kernel 1010 while the filter kernel 1010 moves according to the stride until it reaches the last sample of the input data 1005. , a feature map 1030 having a predetermined size can be obtained.

The convolution layers included in the neural network 240 may be processed according to the convolution operation process described in relation to FIG. 10, but the convolution operation process described in FIG. 10 is only an example, and one embodiment is limited to this. It doesn't work.

Referring again to FIG. 9, the feature maps of the first convolutional layer 910 may be input to the first activation layer 920.

The first activation layer 920 may provide non-linear characteristics to each feature map. The first activation layer 920 may include a sigmoid function, a Tanh function, a Rectified Linear Unit (ReLU) function, etc., but the embodiment is not limited thereto.

Giving non-linear characteristics in the first activation layer 920 may mean changing and outputting some sample values of feature maps. At this time, the change can be performed by applying non-linear characteristics.

The first activation layer 920 may determine whether to transfer sample values of the feature map to the second convolution layer 930. For example, among the sample values of the feature map, some sample values are activated by the first activation layer 920 and transmitted to the second convolution layer 930, and some sample values are activated by the first activation layer 920. It may be deactivated and not transmitted to the second convolution layer 930. Unique characteristics of the input data represented by the feature maps may be emphasized by the first activation layer 920.

The feature maps 925 output from the first activation layer 920 may be input to the second convolution layer 930. One of the feature maps 925 shown in FIG. 9 may correspond to the result of processing the feature map 1030 described with reference to FIG. 10 in the first activation layer 920.

32 The output of the second convolution layer 930 may be input to the second activation layer 940. The second activation layer 940 may provide non-linear characteristics to the input feature maps.

The feature maps 945 output from the second activation layer 940 may be input to the third convolution layer 950. 32 do.

9 shows that the neural network 240 has three convolutional layers (first convolutional layer 910, second convolutional layer 930, and third convolutional layer 950) and two activation layers (first activation layer). Although it is shown as including a layer 920 and a second activation layer 940, this is only an example. In one embodiment, the number of convolution layers and activation layers included in the neural network 240 is It can be changed in various ways.

In one embodiment, the neural network 240 may be implemented through a recurrent neural network (RNN). This may mean changing the CNN structure of the neural network 240 shown in FIG. 9 to an RNN structure.

In one embodiment, the image decoding device 100 and the image encoding device 1900 described later may include at least one Arithmetic logic unit (ALU) for the above-described convolution operation and activation layer operation.

ALU can be implemented as a processor. For the convolution operation, the ALU may include a multiplier that performs a multiplication operation between sample values of input data or a feature map output from the previous layer and sample values of the filter kernel, and an adder that adds the resultant values of the multiplication.

For the operation of the activation layer, the ALU has a multiplier that multiplies the input sample value by a weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and compares the multiplication result with a predetermined value to transfer the input sample value to the next layer. It may include a comparator that determines whether to forward it to .

Previously, according to the convolution operation process shown in FIG. 10, a 5X5-sized filter kernel 1010 is applied to 7X7-sized input data 1005, thereby obtaining a 3X3-sized feature map 1030.

In general, when convolution processing is performed on unpadded input data, output data of a size smaller than the input data is output. Therefore, in order to match the size of the final prediction block to the size of the current block, padding needs to be performed on the input data.

In the embodiment shown in FIG. 10, in order to obtain the feature map 1030 of the same size as the size of the input data 1005, 2 Padding should be equal to the distance.

In one embodiment, the neural network setting unit 230 pads at least one of a preliminary prediction block, a quantization error map, or a POC map to obtain a final prediction block of the same size as the current block, and uses the extended prediction block obtained through padding. At least one of a preliminary prediction block, an extended quantization error map, or an extended POC map may be input to the neural network 240.

In one embodiment, before the convolution operation is performed on the input data or the data output from the previous layer in each convolution layer of the neural network 240, the input data or the data output from the previous layer are padded so that the convolution operation before the convolution operation is performed. The size of the data and the size of the data after the convolution operation may remain the same. Accordingly, when at least one of an extended preliminary prediction block, an extended quantization error map, or an extended POC map is input to the neural network 240, the size of the final prediction block output from the neural network 240 is the extended preliminary prediction block, It may be identical to at least one of an extended quantization error map or an extended POC map. In this case, in one embodiment, the final prediction block may be cropped so that the size of the final prediction block output from the neural network 240 is the same as the size of the current block.

In one embodiment, the neural network setting unit 230 may calculate an extension distance for padding based on the number of convolutional layers included in the neural network 240, the size and stride of the filter kernel used in the convolutional layer. .

In one embodiment, the size of the kernel used in each convolution layer included in the neural network 240 is k _i (i=0, 1, ..., L-1), and the size of the kernel in each convolution layer is k i (i=0, 1, ..., L-1). When the stride is s _i (i=0, 1, ..., L-1), the extension distance can be calculated according to Equation 3 below.

[Equation 3]

In Equation 3, h is the expansion distance in the horizontal direction, v is the expansion distance in the vertical direction, M is the horizontal size of the input data, and N is the vertical size of the input data.

If the size of the filter kernels used in the convolutional layers included in the neural network 240 are all the same as k, and the strides in the convolutional layers are all s, Equation 3 can be changed to Equation 4 below: there is.

[Equation 4]

The neural network setting unit 230 determines the horizontal extension distance and the vertical extension distance for padding of the preliminary prediction block, quantization error map, and POC map based on Equation 3 or Equation 4, and sets the extension distance to the extension distance. An extended preliminary prediction block, an extended quantization error map, and an extended POC map that further include corresponding samples may be obtained.

In one embodiment, when the horizontal extension distance and the vertical extension distance are 1, the neural network setting unit 230 adds samples by an extension distance of 1 in the left and right directions, respectively, from the preliminary prediction block, and An extended preliminary prediction block can be obtained by adding samples by an extension distance of 1 in each of the upper and lower directions of the prediction block.

In one embodiment, the neural network setting unit 230 sets a predetermined value outside the boundaries of the preliminary prediction block, the quantization error map, and the POC map to obtain the extended preliminary prediction block, the extended quantization error map, and the extended POC map. Samples can be added. The predetermined value may be 0.

In one embodiment, considering that the preliminary prediction block is obtained from a reference block that is part of the reference image, the neural network setting unit 230 pads the preliminary prediction block, quantization error map, and POC map by using neighboring samples of the reference block. You may also consider them. That is, by using samples adjacent to the reference block instead of padding the preliminary prediction block according to a predetermined sample value, the spatial characteristics of the reference image can be taken into consideration when inter-predicting the current block.

FIG. 11 is a diagram illustrating a method of obtaining an extended preliminary prediction block according to an embodiment.

As shown in FIG. 11, when the horizontal extension distance is h and the vertical extension distance is v, the neural network setting unit 230 selects samples adjacent to the reference block 1110 in the reference image 1100. An extended preliminary prediction block including samples 1120 corresponding to the middle extended distance h and samples of the preliminary prediction block can be obtained. Accordingly, the horizontal distance of the extended preliminary prediction block may be greater than the preliminary prediction block by 2h, and the vertical distance of the extended preliminary prediction block may be greater than the preliminary prediction block by 2v.

FIG. 12 is a diagram illustrating a method of obtaining an extended preliminary prediction block when the boundary of the reference block 1110 corresponds to the boundary of the reference image 1100, according to an embodiment.

When the horizontal extension distance and the vertical extension distance are determined to be 3, the neural network setting unit 230 configures the peripheral samples located within the extension distance of 3 among the peripheral samples located outside the boundary of the reference block 1110 and An extended preliminary prediction block including samples within the preliminary prediction block can be obtained.

As shown in FIG. 12, when the reference image 1100 is divided into six blocks (1210, 1220, 1230, 1240, 1250, and 1110), and the block located in the upper center is the reference block 1110, the neural network The setting unit 230 is located within the left block 1210 of the reference block 1110, surrounding samples located within an extended distance of 3 from the left boundary of the reference block 1110, and the right block 1250 of the reference block 1110. ) and surrounding samples located within an extended distance of 3 from the right border of the reference block 1110, and located within the lower block 1230 of the reference block 1110 and within an extended distance of 3 from the lower border of the reference block 1110. Neighboring samples located within the extended distance can be selected. At this time, in order to determine the rectangular extended preliminary prediction block, the surrounding samples located within the lower left block 1220 of the reference block 1110 and the surrounding samples located within the lower right block 1240 of the reference block 1110 may also be selected.

In one embodiment, when the boundary of the reference block 1110 coincides with the boundary of the reference image 1100, for example, as shown in FIG. 10, the upper boundary of the reference block 1110 matches the boundary of the reference image 1100. ), the surrounding samples located outside the upper boundary of the reference block 1110 are not included in the reference image 1100. Accordingly, the neural network setting unit 230 uses samples in the reference image 1100 that are closest to each of the surrounding samples 1260 located outside the upper boundary of the reference block 1110 to determine the upper boundary of the reference block 1110. Surrounding samples 1260 located outside can be determined.

Referring to FIG. 10, the values of the neighboring samples located in the leftmost row among the neighboring samples 1260 located outside the upper boundary of the reference block 1110 are determined by the closest sample value a in the adjacent block 1210. , the values of the neighboring samples located in the rightmost column among the neighboring samples 1260 may be determined as the nearest sample value k within the adjacent block 1250.

The neural network setting unit 230 applies an extended preliminary prediction block of 11x11 size, which is larger than the reference block 1110 (and preliminary prediction block) of 5x5 size, to the neural network 240, and applies an extended preliminary prediction block of 11x11 size, which is the same size as the current block, to the neural network 240. A final prediction block of 5x5 can be obtained.

Since the size of the quantization error map input to the neural network 240 must be the same as the size of the extended preliminary prediction block, when the extended preliminary prediction block is input to the neural network 240, the neural network setting unit 230 sets the extended preliminary prediction block. An extended quantization error map with the same size as the prediction block can be obtained. This will be explained with reference to FIG. 13.

FIG. 13 is a diagram illustrating a method of obtaining an extended quantization error map 1300 according to an embodiment.

The left side of FIG. 13 shows a quantization error map 530-4 including four samples, and the right side of FIG. 13 shows a quantization error map 1300 expanded according to an expansion distance of 1.

When the quantization parameters for the four samples included in the reference block are a1, b1, c1, and a1, respectively, the first to fourth samples 1301 to 1304 in the quantization error map 530-4 are a2 and b2. , may have sample values of c2 and a2.

When the samples in the preliminary prediction block and the surrounding samples adjacent to the reference block form an extended preliminary prediction block, the quantization error map 530-4 is changed according to the quantization parameter for the surrounding samples located outside the boundary of the reference block. Values of external surrounding samples may be determined.

As shown on the right side of FIG. 13, the left sample 1310 of the first sample 1301 may have a value of e2, and the left sample 1315 of the third sample 1303 may have a value of a2. Here, e2 and a2 are quantization parameters for the sample located to the left of the sample corresponding to the first sample 1301 in the reference block and quantization parameters for the sample located to the left of the sample corresponding to the third sample 1303. Each can be calculated from .

Additionally, the lower left sample 1320 of the third sample 1303 may have a value of f2, the lower sample 1325 of the third sample 1303 may have a value of c2, and the lower left sample 1320 of the third sample 1303 may have a value of c2. The lower sample 1330 may have a value of a2. Additionally, the lower right sample 1335 of the fourth sample 1304 has a value of e2, the right sample 1340 of the fourth sample 1304 has a value of d2, and the right sample of the second sample 1302 (1345) can have the value of e2.

As described with reference to FIG. 12, when the boundary (e.g., upper boundary) of the reference block 1110 corresponds to the boundary of the reference image 1100, surrounding samples located outside the boundary of the reference block 1110 s 1260 may be determined from the nearest samples for which the corresponding neighboring samples 1260 are available. Likewise, the values of neighboring samples located outside the boundary of the quantization error map 530-4 may also be determined from the nearest samples available to the corresponding neighboring samples.

When the upper boundary of the reference block corresponds to the boundary of the reference image, as shown on the right side of FIG. 13, samples 1305, 1360, and 1355 located outside the upper boundary of the quantization error map 530-4. 1350) may have the values e2, a2, b2, and e2. That is, the sample 1305 located at the upper left of the first sample 1301 is determined from the closest left sample 1310 of the first sample 1301, and the sample located at the upper left of the first sample 1301 ( 1360) can be determined from the closest first sample 1301. In addition, the sample 1355 located at the top of the second sample 1302 is determined from the nearest second sample 1302, and the sample 1350 located at the upper right of the second sample 1302 is determined from the closest second sample 1302. 2 can be determined from the sample 1345 to the right of sample 1302.

In one embodiment, the sample values in the POC map correspond to the POC difference between the current image and the reference image, so when the POC map is padded according to the extension distance, the sample values in the extended POC map all correspond to the POC difference between the current image and the reference image. It can have difference values.

Meanwhile, in one embodiment, if there are samples restored before the current block in the current image, blocks containing the samples may be further input to the neural network 240, with reference to FIGS. 14 and 15. Explain.

FIG. 14 is a diagram illustrating reconstructed and unrestored samples in the current image 1400 according to an embodiment.

The current image 1400 can be restored according to a predetermined scanning order. For example, blocks and samples in the current image 1400 may be restored according to raster scan.

As shown in FIG. 14, when the current image 1400 is restored according to raster scan, the samples located on the left of the current block 1410 and the samples located on top of the current block 1410 are the current block ( The restoration may have been completed before the restoration of 1410).

Samples whose restoration was completed before the current block 1410 may provide the neural network 240 with useful information for generating a final prediction block similar to the current block 1410 in that they are spatially adjacent to the current block.

In one embodiment, when the neural network setting unit 230 inputs at least one of an extended preliminary prediction block, an extended quantization error map, or an extended POC map to the neural network 240, the extended preliminary prediction block, the extended quantization An extended current reconstruction block having the same size as at least one of the error map or the extended POC map may be input to the neural network 240 together.

The extended current reconstruction block may include samples corresponding to the extended distance among samples reconstructed before the current block 1410.

Referring to FIG. 14, when the extension distance is 1, the neural network setting unit 230 selects samples 1420 located at an extension distance 1 from the boundary of the current block 1410 among the samples restored before the current block 1410. ) can be used to obtain the extended current restored block.

In FIG. 14 , the samples 1420 corresponding to an extension distance of 1 in the left and upper directions of the current block 1410 have already been restored, but the samples in the current block 1410 and the samples in the right direction of the current block 1410 And since the samples 1430 corresponding to the extension distance 1 in the downward direction have not been reconstructed, the neural network setup unit 230 may obtain these samples for which reconstruction has not been completed from the extended preliminary prediction block. Refer to Figure 15 for this.

FIG. 15 is a diagram illustrating a method of obtaining an extended current restoration block 1500 according to an embodiment.

Referring to FIG. 15, samples 1420 whose reconstruction was completed before the current block 1410 and samples corresponding to the samples 1420 restored before the current block among the samples of the extended preliminary prediction block 1150. An extended current reconstruction block 1500 containing samples 1125 other than these may be obtained.

In one embodiment, when there are two extended preliminary prediction blocks as the current block is bi-predicted, the extended preliminary prediction block 1150 shown in FIG. 15 is the average of the sample values of the two extended preliminary prediction blocks. You can include values as sample values.

As the expanded current reconstruction block 1500 along with the above-described extended preliminary prediction block, extended quantization error map, and extended POC map are processed by the neural network 240, spatial characteristics within the current image can also be considered.

In one embodiment, the neural network setting unit 230 may select one of a plurality of weight sets according to a predetermined criterion and allow the neural network 240 to operate according to the selected weight set.

Each of the plurality of weight sets may include weights used in the calculation process of the layer included in the neural network 240.

In one embodiment, the neural network setting unit 230 selects the size of the current block, the prediction direction of the current block, the quantization parameter for the reference block, the layer to which the current image belongs in the hierarchical structure of the image, or information obtained from the bitstream. Based on at least one of the plurality of weight sets, a weight set to be used to obtain the final prediction block may be selected.

Referring to FIG. 16, the neural network setting unit 230 sets the weight set indicated by the index obtained from the bitstream among the A weight set to the C weight set in the neural network 240 to obtain a final prediction block more similar to the current block. You can.

Figure 16 illustrates that a weight set is selected based on information obtained from the bitstream, but this corresponds to an embodiment. In one embodiment, a weight set to be used to obtain the final prediction block from among a plurality of weight sets may be selected according to a comparison result between the size of the current block and a predetermined threshold.

For example, if the size of the current block is greater than or equal to 64X64, then the C weight set is selected; if the size of the current block is greater than or equal to 16 can be selected.

Also, for example, if the current image corresponds to layer 1 in the image hierarchical structure, the A weight set is selected, if the current image corresponds to layer 2, the B weight set is selected, and if the current image corresponds to layer 3, then the A weight set is selected. A set of C weights may be selected.

Each of the plurality of weight sets may be generated as a result of training for the neural network 240. For example, the A weight set, B weight set, and C weight set shown in FIG. 16 may be obtained by training the neural network 240 according to different training purposes. Setting a different training purpose may mean changing the type of training image used to train the neural network 240 or calculating loss information in a different way.

As an example, in the training process of the neural network 240 shown in FIG. 23, which will be described later, loss information 2306 corresponding to the difference between the final prediction block for training 2305 and the current block for training 2301 may be used. , Since there are various methods for calculating the difference between two blocks, a weight set A is generated by training the neural network 240 based on the loss information calculated according to the first method, and the A weight set is applied to the loss information calculated according to the second method. A B weight set can be generated by training the neural network 240 based on the B weight set. Additionally, a C weight set can be generated by training the neural network 240 based on the loss information calculated according to the third method.

In one embodiment, the neural network setting unit 230 selects a neural network to be used to obtain the final prediction block among a plurality of neural networks, and applies input data (e.g., preliminary prediction block) to the selected neural network to obtain the final prediction block of the current block. Prediction blocks can also be obtained. A plurality of neural networks may be included in the AI-based prediction decoder 132.

At least one of the type of layer, number of layers, size of filter kernel, or stride of the plurality of neural networks may be different from each other.

In one embodiment, the neural network setting unit 230 selects the size of the current block, the prediction direction of the current block, the quantization parameter for the reference block, the layer to which the current image belongs in the hierarchical structure of the image, or information obtained from the bitstream. Based on at least one of the plurality of neural networks, a neural network to be used to obtain the final prediction block may be selected.

In one embodiment, the neural network setting unit 230 performs the neural network-based prediction described above based on at least one of information obtained from the bitstream, the prediction direction of the current block, or whether the extended preliminary prediction block exceeds the boundary of the reference image. You can decide whether to apply the mode or not.

For example, when information obtained from the bitstream indicates that the neural network-based prediction mode is not applied, the preliminary prediction block obtained by the prediction block acquisition unit 220 may be transmitted to the restoration unit 134.

Additionally, for example, if the extended preliminary prediction block is outside the boundary of the reference image, the neural network setting unit 230 determines that the neural network-based prediction mode is not applied to the current block, and the extended preliminary prediction block is used as a reference image. When located within an image, for example, if the boundary of the reference block does not correspond to the boundary of the reference image, it may be determined that the neural network-based prediction mode is applied to the current block.

Additionally, for example, if the prediction direction of the current block is bidirectional, that is, if a motion vector for list 0 and a motion vector for list 1 are obtained for the current block, the neural network-based prediction mode is used for the current block. If it is determined to be applied, and the prediction direction of the current block is unidirectional, it may be determined that the neural network-based prediction mode is not applied to the current block.

FIG. 17 is a diagram illustrating an image decoding method using the image decoding device 100 according to an embodiment.

In step S1710, the video decoding device 100 obtains the motion vector of the current block.

In one embodiment, the video decoding apparatus 100 may obtain the motion vector of the current block according to a rule-based prediction mode. Rule-based prediction modes include merge mode, skip mode, advanced motion vector prediction (AMVP) mode, bi-directional optical flow (BDOF) mode, bi-prediction with CU-level weights (BCW), or decoder-side motion (DMVR). vector refinement) mode may be included. Which of several rule-based prediction modes should be used to obtain the motion vector of the current block can be determined based on prediction mode information included in the bitstream.

In one embodiment, the video decoding apparatus 100 may use information included in the bitstream, for example, a flag or index, to obtain the motion vector of the current block.

In one embodiment, in order to obtain the motion vector of the current block, the video decoding apparatus 100 builds a motion vector candidate list including the motion vectors of the spatial neighboring blocks and/or the temporal neighboring blocks of the current block as motion vector candidates. You may.

In step S1720, the image decoding apparatus 100 obtains a preliminary prediction block using the reference image of the current block and the motion vector of the current block.

In one embodiment, the image decoding apparatus 100 may obtain a preliminary prediction block using a reference block indicated by the motion vector of the current block in the reference image.

In one embodiment, the process of obtaining a preliminary prediction block similar to the current block from a reference image using a motion vector may be referred to as a motion compensation process.

In one embodiment, the preliminary prediction block may correspond to the result of applying interpolation to the reference block indicated by the motion vector of the current block in the reference image. Accordingly, the preliminary prediction block may include subpixels obtained by applying filtering to integer pixels.

In step S1730, the image decoding device 100 applies at least one of a POC map including the POC difference between the current image including the current block and the reference image, a preliminary prediction block, or a quantization error map to the neural network 240 to obtain the current image. The final prediction block of the block can be obtained.

In one embodiment, sample values of the quantization error map may be calculated from the quantization parameters for the reference block.

In one embodiment, the image decoding apparatus 100 pads at least one of a POC map, a preliminary prediction block, or a quantization error map, and uses an extended POC map, an extended preliminary prediction block, or an extended quantization error map obtained through padding. At least one of these can be applied to the neural network 240 to obtain the final prediction block of the current block. In one embodiment, the image decoding apparatus 100 may further input the expanded current reconstruction block to the neural network 240.

In step S1740, the image decoding device 100 restores the current block using the final prediction block and the residual block obtained from the bitstream.

In one embodiment, the image decoding apparatus 100 may obtain the current block by adding sample values of the final prediction block and sample values of the residual block.

In one embodiment, the image decoding apparatus 100 may obtain information about a quantized transform coefficient from a bitstream and obtain a residual block by applying inverse quantization and inverse transform to the quantized transform coefficient.

The image including the restored current block can be used to decode the next block.

Figure 18 is a diagram illustrating syntax according to one embodiment.

In one embodiment, the neural network-based prediction mode described above may be used in conjunction with skip mode, merge mode, or AMVP mode.

Referring to FIG. 18, in S1810, when skip mode is applied to the current block (inter_skip=1), merge_data() is called, and NNinter() is called in S1840 within merge_data(), thereby creating a neural network-based network for the current block. A prediction mode of may be applied.

Additionally, in S1820, if the merge mode is applied to the current block (merge_flag=1), merge_data() is called, and NNinter() is called in S1840 within merge_data(), thereby establishing a neural network-based prediction mode for the current block. It can be applied.

Additionally, in S1830, when AMVP mode is applied to the current block, NNinter() is called so that a neural network-based prediction mode can be applied to the current block.

According to one embodiment, the motion vector of the current block obtained according to the existing skip mode, merge mode, or AMVP mode is adjusted for each sample based on the neural network 240, so that a final prediction block more similar to the current block can be obtained. .

FIG. 19 is a diagram illustrating the configuration of a video encoding device 1900 according to an embodiment.

Referring to FIG. 19, the image encoding device 1900 may include an encoder 1910 and a bitstream generator 1930. The encoder 1910 may include an AI-based prediction encoder 1912 and a residual data acquisition unit 1914.

In one embodiment, the encoder 1910 includes the prediction encoder 2415, transform and quantization unit 2420, inverse quantization and inverse transform unit 2430, deblocking filtering unit 2435, and loop filtering shown in FIG. 24. It can correspond to part 2440. In one embodiment, the bitstream generator 1930 may correspond to the entropy encoder 2425 shown in FIG. 24.

The encoder 1910 and the bitstream generator 1930 may be implemented with at least one processor. The encoder 1910 and the bitstream generator 1930 may operate according to at least one instruction stored in at least one memory.

Although FIG. 19 shows the encoder 1910 and the bitstream generator 1930 separately, the encoder 1910 and the bitstream generator 1930 can be implemented through one processor. In this case, the encoder 1910 and the bitstream generator 1930 may be implemented with a dedicated processor, or a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphic processing unit (GPU) and software. It can also be implemented through a combination of . Additionally, in the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure, or a memory processing unit for using an external memory.

In one embodiment, the encoder 1910 and the bitstream generator 1930 may be comprised of a plurality of processors. In this case, it may be implemented through a combination of dedicated processors, or it may be implemented through a combination of software and multiple general-purpose processors such as AP, CPU, or GPU.

The encoder 1910 may encode the current block using the reference image of the current block. As a result of encoding the current block, information about the residual block and information about the motion vector may be output.

In one embodiment, information about the residual block may not be output by the encoder 1910 according to the rule-based coding mode (eg, skip mode) for the current block.

In one embodiment, the AI-based prediction encoder 1912 may obtain the final prediction block of the current block using a reference image and the current block. The final prediction block may be transmitted to the residual data acquisition unit 1914.

The residual data acquisition unit 1914 may obtain a residual block corresponding to the difference between the current block and the final prediction block.

In one embodiment, the residual data acquisition unit 1914 may obtain a residual block by subtracting sample values of the final prediction block from sample values of the current block.

In one embodiment, the residual data acquisition unit 1914 may obtain quantized transform coefficients by applying transform and quantization to samples of the residual block.

Information about the residual block and information about the motion vector obtained by the encoder 1910 may be transmitted to the bitstream generator 1930.

In one embodiment, the information about the residual block may include information about the quantized transform coefficient (eg, a flag indicating whether the quantized transform coefficient is 0, etc.).

In one embodiment, the information about the motion vector may include information indicating one or more motion vector candidates among the motion vector candidates included in the motion vector candidate list, for example, a flag or an index.

In one embodiment, the information about the motion vector may include a differential motion vector between the motion vector of the current block and the predicted motion vector.

The bitstream generator 1930 may generate a bitstream including the encoding result for the current block.

The bitstream may be transmitted to the video decoding device 100 through a network. In one embodiment, the bitstream is a magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. ) and the like.

In one embodiment, the bitstream generator 1930 may generate a bitstream by entropy coding syntax elements corresponding to the encoding result for the image.

FIG. 20 is a diagram illustrating the configuration of an AI-based predictive encoding unit 1912 according to an embodiment.

Referring to FIG. 20, the AI-based prediction encoding unit 1912 may include a motion information acquisition unit 2010, a prediction block acquisition unit 2020, a neural network setting unit 2030, and a neural network 2040.

Neural network 2040 may be stored in memory. In one embodiment, neural network 2040 may be implemented with an AI processor.

The motion information acquisition unit 2010 may search for a block similar to the current block in the reference image and obtain a motion vector indicating the searched block.

In one embodiment, in order to encode the motion vector of the current block, the motion information acquisition unit 2010 creates a motion vector candidate list that includes the motion vectors of the spatial neighboring blocks and/or the temporal neighboring blocks of the current block as motion vector candidates. It can be built.

In one embodiment, the motion vector acquired by the motion information acquisition unit 2010 may include a motion vector for list 0, a motion vector for list 1, or a motion vector for list 0 and a motion vector for list 1. there is.

The motion vector for list 0 is a motion vector pointing to a reference block in the reference image included in list 0 (or reference image list 0), and the motion vector for list 1 is included in list 1 (or reference image list 1). It may be a motion vector to indicate a reference block in the reference image.

The prediction block acquisition unit 2020 may acquire a preliminary prediction block using the reference block indicated by the motion vector in the reference image.

In one embodiment, the preliminary prediction block may be obtained by applying interpolation to the reference block indicated by the motion vector within the reference image. In this case, the preliminary prediction block may include subpixels obtained through filtering on integer pixels.

In one embodiment, the motion information acquisition unit 2010 and the prediction block acquisition unit 2020 may acquire a preliminary prediction block according to a rule-based prediction mode.

As described above, when the motion vector for list 0 is acquired by the motion information acquisition unit 2010, the prediction block acquisition unit 2020 determines the motion vector indicated by the motion vector for list 0 within the reference image included in list 0. A preliminary prediction block can be obtained using a reference block.

In one embodiment, when a motion vector for List 1 is acquired by the motion information acquisition unit 2010, the prediction block acquisition unit 2020 determines the motion vector pointed to by the motion vector for List 1 within the reference image included in List 1. A preliminary prediction block can be obtained using a reference block.

In one embodiment, when a motion vector for list 0 and a motion vector for list 1 are acquired by the motion information acquisition unit 2010, the prediction block acquisition unit 2020 acquires the list within the reference image included in list 0. Obtain a preliminary prediction block for list 0 using the reference block pointed to by the motion vector for list 0, and obtain a preliminary prediction block for list 1 using the reference block pointed to by the motion vector for list 1 within the reference image included in list 1. A prediction block can be obtained.

The neural network setting unit 2030 may acquire data to be input to the neural network 2040.

In one embodiment, the neural network setting unit 2030 may obtain data to be input to the neural network 2040 based on quantization parameters for the reference image, preliminary prediction block, and reference block.

In one embodiment, the final prediction block of the current block may be obtained by applying at least one of a preliminary prediction block, a POC map, or a quantization error map to the neural network 2040 by the neural network setting unit 2030.

Since the neural network setting unit 2030 and the neural network 2040 may be the same as the neural network setting unit 230 and the neural network 240 included in the AI prediction decoding unit 132 shown in FIG. 2, detailed descriptions are omitted. .

In one embodiment, the neural network setting unit 2030 selects one of different quantization error map acquisition methods (e.g., the quantization error map acquisition methods shown in FIGS. 5 to 8) and selects the selected A quantization error map can be obtained depending on the method.

In one embodiment, the neural network setup unit 2030 uses any one of different quantization error map acquisition methods (e.g., the quantization error map acquisition methods shown in FIGS. 5 to 8) based on the cost. You can choose. When calculating cost, rate-distortion cost can be used.

In one embodiment, the motion information acquisition unit 2010 changes the precision of the motion vector of the current block from fraction precision to integer precision, and the prediction block acquisition unit 2020 changes the precision of the motion vector of the current block to integer precision. Accordingly, a preliminary prediction block may be obtained. In one embodiment, a reference block pointed to by an integer precision motion vector within a reference image may be determined as a preliminary prediction block.

As the precision of the motion vector of the current block is changed to integer precision, the number of bits for expressing the motion vector of the current block may be reduced, and thus the bit rate of the bitstream may be reduced.

Another reason for changing the precision of the motion vector of the current block from fractional precision to integer precision is to intentionally provide information about the less accurate motion vectors to the neural network 2040, so that the neural network 2040 can change the precision from inaccurate motion vectors to accurate motion vectors. This is to be able to derive the motion vector on its own.

Change in motion vector precision according to one embodiment may be related to AMVR (Adaptive motion vector resolution) mode included in the VVC standard. AMVR mode is a mode that adaptively selects and uses the resolution of motion vectors and residual motion vectors.

In AMVR mode, motion vectors and residual motion vectors can generally be encoded/decoded using any one of the resolutions of 1/4pel, 1/2pel, 1pel, and 4pel.

If the current block is encoded/decoded using a 1pel or 4pel motion vector in AMVR mode, the final prediction block can be expected to be the same as the one generated based on a higher resolution motion vector through the neural network-based prediction mode.

In addition, when the current block is encoded/decoded using a motion vector of 1/4pel or 1/2pel in AMVR mode, in neural network-based prediction mode, the motion vector is signaled by changing 1/4pel or 1/2pel to 1pel. The number of bits required can be reduced, and a high quality final prediction block can be obtained through processing of the preliminary prediction block by the neural network 2040.

A method by which the motion information acquisition unit 2010 changes the precision of the motion vector of the current block from fractional precision to integer precision will be described with reference to FIG. 21.

FIG. 21 is a diagram illustrating a method of changing a fractional precision motion vector to an integer precision motion vector according to an embodiment.

In one embodiment, the motion information acquisition unit 2010 determines the motion vector (A) of the current block when the precision of the motion vector (A) is fractional precision (e.g., 1/2pe, 1/4pel, or 1/8pel, etc.). The motion vector (A) can be changed to point to integer pixels.

In one embodiment, in Figure 21, it is assumed that the motion vector (A) points to coordinates (19/4, 27/4) (2110) based on coordinates (0,0). Since the coordinates (19/4, 27/4) 2110 are not integer pixels, the motion information acquisition unit 2010 can adjust the motion vector (A) to point to integer pixels.

The coordinates of surrounding integer pixels centered on coordinates (19/4, 27/4)(2110) are (16/4, 28/4)(2130), (16/4, 24/4)(2120), respectively. , (20/4, 28/4)(2140), (20/4, 24/4)(2150). At this time, the motion information acquisition unit 2010 uses the coordinates (20/4, 28/4) (2140) where the motion vector (A) is located in the upper right corner instead of the coordinates (19/4, 27/4) (2110). ) can be changed to point to .

In one embodiment, the motion information acquisition unit 2010 determines the motion vector (A) at coordinates 2120 located at the bottom left, coordinates 2130 located at the top left, or coordinates 2150 located at the bottom right. ) can also be changed to point to .

The process of changing the fractional precision of the motion vector (A) to integer precision can be referred to as rounding the motion vector.

FIG. 22 is a diagram illustrating an image encoding method using the image encoding device 1900 according to an embodiment.

In step S2210, the image encoding device 1900 obtains the motion vector of the current block using the reference image.

In one embodiment, the video encoding apparatus 1900 may obtain a motion vector indicating a block similar to the current block within the reference video.

In one embodiment, the process of acquiring the motion vector of the current block using a reference image may be referred to as a motion prediction process.

In step S2220, the image encoding device 1900 obtains a preliminary prediction block using the reference image of the current block and the motion vector of the current block.

In one embodiment, the image encoding apparatus 1900 may obtain a preliminary prediction block using the reference block indicated by the motion vector of the current block in the reference image.

In one embodiment, the process of obtaining a preliminary prediction block similar to the current block from a reference image using a motion vector may be referred to as a motion compensation process.

In one embodiment, the preliminary prediction block may correspond to the result of applying interpolation to the reference block indicated by the motion vector of the current block in the reference image.

In step S2230, the image encoding device 1900 sends at least one of a POC map, a preliminary prediction block, or a quantization error map including the POC difference between the current image including the current block and the reference image as sample values to the neural network 2040. By applying it, you can obtain the final prediction block of the current block.

In one embodiment, sample values of the quantization error map may be calculated from the quantization parameters for the reference block.

In one embodiment, the image encoding device 1900 pads at least one of a POC map, a preliminary prediction block, or a quantization error map, and uses an extended POC map, an extended preliminary prediction block, or an extended quantization error map obtained through padding. At least one of these can be applied to the neural network 2040 to obtain the final prediction block of the current block. In this case, in one embodiment, the image encoding apparatus 1900 may further input the expanded current reconstruction block to the neural network 2040.

In step S2240, the image encoding device 1900 obtains a residual block using the current block and the final prediction block.

In one embodiment, the image encoding apparatus 1900 may obtain a residual block by subtracting sample values of the final prediction block from sample values of the current block.

In one embodiment, the image encoding apparatus 1900 may obtain quantized transform coefficients by applying transform and quantization to samples of the residual block.

In step S2250, the image encoding device 1900 may generate a bitstream including information about the residual block.

In one embodiment, the information about the residual block may include information about the quantized transform coefficient (eg, a flag indicating whether the quantized transform coefficient is 0, etc.).

In one embodiment, the bitstream may further include information about the motion vector of the current block. Information about the motion vector of the current block may include information indicating one or more motion vector candidates among the motion vector candidates included in the motion vector candidate list, for example, a flag or an index.

In one embodiment, the information about the motion vector may include a differential motion vector between the motion vector of the current block and the predicted motion vector.

In one embodiment, the bitstream may not include information about residual blocks. For example, if the rule-based prediction mode applied to the current block is skip mode, the bitstream may not include information about the residual block for the current block.

Hereinafter, a training method of the neural network 240 used by at least one of the image decoding apparatus 100 or the image encoding apparatus 1900 will be described with reference to FIG. 23.

FIG. 23 is a diagram for explaining a training method of the neural network 240 according to an embodiment.

The current block for training 2301 shown in FIG. 23 may correspond to the current block described above. Additionally, the preliminary prediction block for training 2302, the quantization error map for training 2303, and the POC map for training 2304 may correspond to the preliminary prediction block, quantization error map, and POC map shown in FIGS. 2 and 20. . The preliminary prediction block for training 2302 may correspond to a block identified in the reference image for training by the motion vector of the current block for training 2301.

According to the training method of the neural network 240 according to the present disclosure, the neural network 240 is trained so that the final prediction block 2305 for training output from the neural network 240 is the same or similar to the current block 2301 for training. . To this end, loss information 2306 corresponding to the difference between the final prediction block for training 2305 and the current block for training 2301 may be used for training the neural network 240.

When describing the training process of the neural network 240 in detail with reference to FIG. 23, first, the preliminary prediction block for training 2302, the quantization error map for training 2303, and the POC map for training 2304 are transferred to the neural network 240. input, and the final prediction block 2305 for training may be output from the neural network 240. The neural network 240 may operate according to preset weights.

Loss information 2306 corresponding to the difference between the final prediction block for training 2305 and the current block for training 2301 is calculated, and the weight set in the neural network 240 can be updated according to the loss information 2306. . The neural network 240 may update the weights so that the loss information 2306 is reduced or minimized.

The loss information 2306 includes the L1-norm value, L2-norm value, Structural Similarity (SSIM) value, and Peak Signal Signal (PSNR-HVS) for the difference between the current block for training (2301) and the final prediction block for training (2305). -It may include at least one of a To-Noise Ratio-Human Vision System) value, a Multiscale SSIM (MS-SSIM) value, a Variance Inflation Factor (VIF) value, or a Video Multimethod Assessment Fusion (VMAF) value.

Training of the neural network 240 according to one embodiment may be performed by a training device. The training device may be the video decoding device 100 or the video encoding device 1900. Depending on the implementation, the training device may be an external server. In this case, the neural network 240 and weights trained by an external server may be transmitted to the image decoding device 100 and the image encoding device 1900.

The video decoding device 100 and video encoding device 1900 using AI according to an embodiment, and the method using them, produce a final prediction block 955 that is more similar to the current block 300; 1410 compared to the existing rule-based prediction mode. The task is to obtain.

In addition, the image decoding apparatus 100 and the image encoding apparatus 1900 using AI according to an embodiment, and the method thereof have the task of reducing the bit rate of a bitstream including information about a residual block.

A method of decoding an image according to an embodiment may include obtaining a motion vector of the current block (300; 1410) (S1710).

In one embodiment, the image decoding method uses a reference block (415;435;510;510-1;510-2;1110) indicated by a motion vector within the reference image (410;430;1100) to make preliminary prediction. It may include obtaining the block 902 (S1720).

In one embodiment, the image decoding method includes a Picture Order Difference (POC) difference between the current image (400;1400) including the current block (300;1410) and the reference image (410;430;1100). At least one of the POC map 906, the preliminary prediction block 902, or the quantization error map 530;530-1;530-2;530-3;530-4;904 is applied to the neural network 240 to determine the current block. It may include a step (S1730) of obtaining the final prediction block 955 of (300;1410).

In one embodiment, the image decoding method may include restoring the current block 300 (1410) using the residual block and the final prediction block 955 obtained from the bitstream (S1740).

In one embodiment, the sample values of the quantization error map (530;530-1;530-2;530-3;530-4;904) are the reference block (415;435;510;510-1;510-2) ;1110) can be calculated from the quantization parameters.

In one embodiment, the sample values of the quantization error map (530;530-1;530-2;530-3;530-4;904) may correspond to the quantization step size or quantization error value calculated from the quantization parameter. .

In one embodiment, the quantization error map (530;530-1;530-2;530-3;530-4;904) is the reference block (415;435;510;510-1;510-2;1110) can be divided into sub-areas corresponding to the sub-blocks, and the sample value included in each of the sub-areas of the quantization error map (530;530-1;530-2;530-3;530-4;904) may be calculated from quantization parameters for samples at a certain location within a sub-block of the reference block (415; 435; 510; 510-1; 510-2; 1110).

In one embodiment, obtaining the final prediction block 955 of the current block 300 (1410) includes using an enlarged POC map, an enlarged preliminary prediction block 1150, or an enlarged quantization error map 1300. It may include obtaining a final prediction block 955 of the current block 300 (1410) by applying at least one of them to the neural network 240.

In one embodiment, at least one of the extended POC map, extended preliminary prediction block 1150, or extended quantization error map 1300 includes POC map 906, preliminary prediction block 902, or quantization error map 530; At least one of 530-1;530-2;530-3;530-4;904) may be padded according to the extension distance.

In one embodiment, neural network 240 may include one or more convolutional layers.

In one embodiment, the image decoding method determines the expansion distance based on the number of convolutional layers included in the neural network 240, the size and stride of the filter kernel used in the convolutional layer, and determines the expansion distance based on the reference image 410; Among the neighboring samples outside the boundary of the reference block (415;435;510;510-1;510-2;1110) within 430;1100), the neighboring samples corresponding to the extension distance and the samples of the preliminary prediction block 920 It may include obtaining an extended preliminary prediction block 1150 that includes.

In one embodiment, the image decoding method includes quantization parameters for neighboring samples corresponding to the extension distance within the reference image (410;430;1100) and reference blocks (415;435;510;510-1;510-2); It may include obtaining an extended quantization error map 1300 including sample values calculated from quantization parameters for 1110).

In one embodiment, if the boundary of the reference block (415;435;510;510-1;510-2;1110) corresponds to the boundary of the reference image (410;430;1100), the surrounding samples corresponding to the extension distance are , can be determined from the closest available sample in the reference image (410;430;1100).

In one embodiment, obtaining the final prediction block 955 of the current block 300; 1410 includes at least one of an extended POC map, an extended preliminary prediction block 1150, or an extended quantization error map 1300. It may include applying the expanded current restoration block 1500 to the neural network 240.

In one embodiment, the extended current reconstruction block 1500 includes neighboring samples 1420 reconstructed before the current block 300; 1410 in the current image 400; 1400, and the extended preliminary prediction block 1150. ) may include samples 1125 other than the samples corresponding to the surrounding samples 1420 restored before the current block 300 (1410).

In one embodiment, the image decoding method includes the size of the current block (300;1410), the prediction direction of the current block (300;1410), and the reference block (415;435;510;510-1;510-2;1110). ), based on at least one of the quantization parameters for, the layer to which the current image (400; 1400) belongs in the hierarchical structure of the image, or information obtained from the bitstream, obtain the final prediction block 955 from a plurality of weight sets. It may include selecting a set of weights to be used.

In one embodiment, the final prediction block 955 may be obtained based on the neural network 240 operating according to the selected weight set.

An image encoding method according to an embodiment includes reference blocks (415;435;510;510-1;510-2;1110) in the reference image (410;430;1100) corresponding to the current block (300;1410). It may include a step (S2210) of acquiring a motion vector pointing to .

In one embodiment, a POC map 906 containing the POC difference between the current image 400;1400 and the reference image 410;430;1100, including the current block 300;1410, the reference block 415; The preliminary prediction block 902 or the quantization error map (530;530-1;530-2;530-3;530-4;904) obtained based on 435;510;510-1;510-2;1110) It may include a step (S2230) of obtaining the final prediction block 955 of the current block 300 (1410) by applying at least one of them to the neural network 2040.

In one embodiment, the video encoding method may include obtaining a residual block using the current block 300 (1410) and the final prediction block 955 (S2240).

In one embodiment, a method of encoding an image may include generating a bitstream including information about information about a residual block (S2250).

In one embodiment, the sample values of the quantization error map (530;530-1;530-2;530-3;530-4;904) are the reference block (415;435;510;510-1;510-2) ;1110) can be calculated from the quantization parameters.

In one embodiment, the video encoding method may include changing the precision of the obtained motion vector from fractional precision to integer precision.

In one embodiment, the reference block (415;435;510;510-1;510-2;1110) indicated by the integer precision motion vector may be determined as the preliminary prediction block 902.

In one embodiment, obtaining the final prediction block 955 of the current block 300; 1410 includes at least one of an extended POC map, an extended preliminary prediction block 1150, or an extended quantization error map 1300. It may include applying to the neural network 2040 to obtain the final prediction block 955 of the current block 300; 1410.

In one embodiment, at least one of the extended POC map, extended preliminary prediction block 1150, or extended quantization error map 1300 includes POC map 906, preliminary prediction block 902, or quantization error map 530; It may be padded according to the extension distance from at least one of (530-1;530-2;530-3;530-4;904).

An image decoding device according to an embodiment includes at least one memory storing at least one instruction; and at least one processor operating according to at least one instruction.

In one embodiment, at least one processor of the video decoding apparatus may obtain a motion vector of the current block 300 (1410).

In one embodiment, at least one processor of the image decoding apparatus selects a reference block (415;435;510;510-1;510-2;1110) indicated by a motion vector within the reference image (410;430;1100). The preliminary prediction block 902 can be obtained using.

In one embodiment, at least one processor of the image decoding apparatus is configured to configure a POC difference between a current image (400;1400) including a current block (300;1410) and a reference image (410;430;1100). At least one of the map 906, the preliminary prediction block 902, or the quantization error map 530;530-1;530-2;530-3;530-4;904 is applied to the neural network 240 to determine the current block ( The final prediction block 955 (300;1410) can be obtained.

In one embodiment, at least one processor of the image decoding apparatus may restore the current block 300 (1410) using the residual block and the final prediction block 955 obtained from the bitstream.

In one embodiment, the sample values of the quantization error map (530;530-1;530-2;530-3;530-4;904) are the reference block (415;435;510;510-1;510-2) ;1110) can be calculated from the quantization parameters.

An image encoding device according to an embodiment includes at least one memory storing at least one instruction; and at least one processor operating according to at least one instruction.

In one embodiment, at least one processor of the image encoding device is configured to control the reference block (415;435;510;510-1;510-) in the reference image (410;430;1100) corresponding to the current block (300;1410). A motion vector pointing to 2;1110) can be obtained.

In one embodiment, the at least one processor of the image encoding device is configured to configure a POC difference between a current image (400;1400) including a current block (300;1410) and a reference image (410;430;1100). The map 906, the preliminary prediction block 902 obtained based on the reference block 415;435;510;510-1;510-2;1110, or the quantization error map 530;530-1;530-2; At least one of 530-3;530-4;904) can be applied to the neural network 2040 to obtain the final prediction block 955 of the current block 300;1410.

In one embodiment, at least one processor of the image encoding device may obtain a residual block using the current block 300 (1410) and the final prediction block 955.

In one embodiment, at least one processor of an image encoding device may generate a bitstream including information about information about a residual block.

In one embodiment, the sample values of the quantization error map (530;530-1;530-2;530-3;530-4;904) are the reference block (415;435;510;510-1;510-2) ;1110) can be calculated from the quantization parameters.

The video decoding device 100 and video encoding device 1900 using AI according to an embodiment, and the method using them, produce a final prediction block 955 that is more similar to the current block 300; 1410 compared to the existing rule-based prediction mode. can be obtained.

In addition, the image decoding apparatus 100 and the image encoding apparatus 1900 using AI according to an embodiment, and the method thereof can reduce the bit rate of a bitstream including information about the residual block.

Meanwhile, the above-described embodiments of the present disclosure can be written as a program that can be executed on a computer, and the written program can be stored in a storage medium that can be read by a device.

A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is semi-permanently stored in a storage medium and temporary storage media. It does not distinguish between cases where it is stored as . For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. A computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store or between two user devices (e.g. smartphones). It may be distributed in person or online (e.g., downloaded or uploaded). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) is stored on a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. It can be temporarily stored or created temporarily.

Above, the technical idea of the present disclosure has been described in detail with preferred embodiments, but the technical idea of the present disclosure is not limited to the above embodiments, and those with ordinary knowledge in the field within the scope of the technical idea of the present disclosure Various modifications and changes are possible depending on the user.

Claims (15)

In the method of decoding a video,
Obtaining the motion vector of the current block (300; 1410) (S1710);
Obtaining a preliminary prediction block 902 using the reference block (415;435;510;510-1;510-2;1110) indicated by the motion vector in the reference image (410;430;1100) (S1720) );
A POC map 906 including a POC (Picture Order Difference) difference between the current image 400; 1400 including the current block 300; 1410 and the reference image 410; 430; 1100, and the preliminary prediction. At least one of the block 902 or the quantization error map (530;530-1;530-2;530-3;530-4;904) is applied to the neural network 240 to determine the final result of the current block (300;1410). Obtaining a prediction block 955 (S1730); and
Comprising a step (S1740) of restoring the current block (300; 1410) using a residual block obtained from a bitstream and the final prediction block (955),
The sample values of the quantization error map (530;530-1;530-2;530-3;530-4;904) are the reference block (415;435;510;510-1;510-2;1110) A method of decoding an image, calculated from quantization parameters for.
According to paragraph 1,
Sample values of the quantization error map (530;530-1;530-2;530-3;530-4;904) are,
A method of decoding an image corresponding to the quantization step size or quantization error value calculated from the quantization parameter.
According to any one of paragraphs 1 and 2,
The quantization error map (530;530-1;530-2;530-3;530-4;904) is a subblock of the reference block (415;435;510;510-1;510-2;1110). It is divided into sub-areas corresponding to the
Sample values included in each of the sub-regions of the quantization error map (530;530-1;530-2;530-3;530-4;904) are the reference block (415;435;510;510-1). ;510-2;1110) A decoding method of an image calculated from the quantization parameter for a sample at a predetermined position in a subblock.
According to any one of claims 1 to 3,
The step of obtaining the final prediction block 955 of the current block 300; 1410 is,
At least one of an enlarged POC map, an expanded preliminary prediction block 1150, or an expanded quantization error map 1300 is applied to the neural network 240 to generate a final prediction block ( 955), including the step of obtaining,
At least one of the extended POC map, the extended preliminary prediction block 1150, or the extended quantization error map 1300 is the POC map 906, the preliminary prediction block 902, or the quantization error map 530. ;530-1;530-2;530-3;530-4;904), wherein at least one of the following is padded according to the extension distance.
According to any one of claims 1 to 4,
The neural network 240 includes one or more convolutional layers,
The method of decoding the video is,
determining the expansion distance based on the number of convolutional layers included in the neural network 240 and the size and stride of a filter kernel used in the convolutional layer; and
Among the surrounding samples outside the boundary of the reference block (415;435;510;510-1;510-2;1110) in the reference image (410;430;1100), the surrounding samples corresponding to the extension distance and the A method for decoding an image, further comprising obtaining the extended preliminary prediction block 1150 including samples of the preliminary prediction block 920.
According to any one of claims 1 to 5,
The method of decoding the video is,
Calculated from the quantization parameters for surrounding samples corresponding to the extension distance in the reference image (410;430;1100) and the quantization parameters for the reference block (415;435;510;510-1;510-2;1110) A method for decoding an image, further comprising obtaining the extended quantization error map 1300 including sample values.
According to any one of claims 1 to 6,
If the boundary of the reference block (415;435;510;510-1;510-2;1110) corresponds to the boundary of the reference image (410;430;1100), the surrounding samples corresponding to the extension distance are A method of decoding an image, determined from the closest available sample in the reference image (410;430;1100).
According to any one of claims 1 to 7,
The step of obtaining the final prediction block 955 of the current block 300; 1410 is,
Applying the extended current reconstruction block 1500 together with at least one of the extended POC map, the extended preliminary prediction block 1150, or the extended quantization error map 1300 to the neural network 240. do,
The extended current reconstruction block 1500 includes neighboring samples 1420 reconstructed before the current block 300; 1410 in the current image 400; 1400, and the extended preliminary prediction block 1150. A method of decoding an image, including samples (1125) other than samples corresponding to neighboring samples (1420) restored before the current block (300; 1410) among the samples.
According to any one of claims 1 to 8,
The method of decoding the video is,
Size of the current block (300;1410), prediction direction of the current block (300;1410), quantization parameters for the reference block (415;435;510;510-1;510-2;1110), image A weight set to be used to obtain the final prediction block 955 among a plurality of weight sets based on at least one of the layer to which the current image (400; 1400) belongs in a hierarchical structure or information obtained from the bitstream. Further comprising the step of selecting,
A method of decoding an image in which the final prediction block 955 is obtained based on the neural network 240 operating according to the selected weight set.
A computer-readable recording medium on which a program for performing the method of any one of claims 1 to 9 is recorded on a computer.
In a method of encoding an image,
Obtaining a motion vector pointing to the reference block (415;435;510;510-1;510-2;1110) in the reference image (410;430;1100) corresponding to the current block (300;1410) (S2210) ;
A POC map 906 including the POC difference between the current image 400;1400 including the current block 300;1410 and the reference image 410;430;1100, the reference block 415;435; At least one of the preliminary prediction block 902 obtained based on 510;510-1;510-2;1110) or the quantization error map (530;530-1;530-2;530-3;530-4;904) Obtaining the final prediction block 955 of the current block 300; 1410 by applying one to the neural network 2040 (S2230);
Obtaining a residual block using the current block (300; 1410) and the final prediction block (955) (S2240); and
A step of generating a bitstream including information about the residual block (S2250),
The sample values of the quantization error map (530;530-1;530-2;530-3;530-4;904) are the reference block (415;435;510;510-1;510-2;1110) An image encoding method calculated from quantization parameters for.
In paragraph 11
The video encoding method is,
Further comprising changing the precision of the obtained motion vector from fractional precision to integer precision,
A video encoding method in which a reference block (415;435;510;510-1;510-2;1110) indicated by the integer precision motion vector is determined as the preliminary prediction block (902).
According to any one of claims 11 and 12,
The step of obtaining the final prediction block 955 of the current block 300; 1410 is,
At least one of an extended POC map, an extended preliminary prediction block 1150, or an extended quantization error map 1300 is applied to the neural network 2040 to produce the final prediction block 955 of the current block 300; 1410. Including the step of obtaining,
At least one of the extended POC map, the extended preliminary prediction block 1150, or the extended quantization error map 1300 is the POC map 906, the preliminary prediction block 902, or the quantization error map 530. ;530-1;530-2;530-3;530-4;904), wherein the video is padded according to an extension distance from at least one of the following.
at least one memory storing at least one instruction; and
At least one processor operating according to the at least one instruction,
The at least one processor,
Obtain the motion vector of the current block (300; 1410),
Obtain a preliminary prediction block 902 using the reference block (415;435;510;510-1;510-2;1110) indicated by the motion vector in the reference image (410;430;1100),
A POC map 906 including the POC difference between the current image 400;1400 including the current block 300;1410 and the reference image 410;430;1100, the preliminary prediction block 902, or At least one of the quantization error maps (530;530-1;530-2;530-3;530-4;904) is applied to the neural network 240 to generate the final prediction block 955 of the current block 300;1410. obtain,
Restore the current block (300; 1410) using the residual block obtained from the bitstream and the final prediction block (955),
The sample values of the quantization error map (530;530-1;530-2;530-3;530-4;904) are the reference block (415;435;510;510-1;510-2;1110) An image decoding device calculated from quantization parameters for.
at least one memory storing at least one instruction; and
At least one processor operating according to the at least one instruction,
The at least one processor,
Obtain a motion vector pointing to the reference block (415;435;510;510-1;510-2;1110) in the reference image (410;430;1100) corresponding to the current block (300;1410),
A POC map 906 including the POC difference between the current image 400;1400 including the current block 300;1410 and the reference image 410;430;1100, the reference block 415;435; At least one of the preliminary prediction block 902 obtained based on 510;510-1;510-2;1110) or the quantization error map (530;530-1;530-2;530-3;530-4;904) Apply one to the neural network 2040 to obtain the final prediction block 955 of the current block 300; 1410,
Obtaining a residual block using the current block (300; 1410) and the final prediction block (955),
Generating a bitstream containing information about information about the residual block,
The sample values of the quantization error map (530;530-1;530-2;530-3;530-4;904) are the reference block (415;435;510;510-1;510-2;1110) An image encoding device calculated from quantization parameters for.

Images