KR20230041601A - Apparatus and method for encoding and decodng image using artificial intelligence - Google Patents

Abstract

According to an embodiment, a method for decoding a video may comprise the steps of: obtaining feature data of a current optical flow and feature data of a current residual image from a bitstream; obtaining the current optical flow and first weight data by applying the feature data of the current optical flow to an optical flow decoder; obtaining the current residual image by applying the feature data of the current residual image to a residual decoder; obtaining a preliminary prediction image from a previous restored image based on the current optical flow; obtaining a final prediction image by applying sample values of the first weight data to sample values of the preliminary prediction image; and obtaining a current restored image corresponding to the current image by combining the final prediction image and the current residual image.

Description

Apparatus and method for encoding and decoding images using AI

The present disclosure relates to encoding and decoding of video. More specifically, the present disclosure relates to a technique for encoding and decoding an image using AI (Artificial Intelligence), eg, a neural network.

In codecs such as H.264 Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC), an image is divided into blocks, and each block is predicted and encoded through inter prediction or intra prediction. Predictive decoding is possible.

Intra prediction is a method of compressing an image by removing spatial redundancy in images, and inter prediction is a method of compressing an image by removing temporal redundancy between images.

As a representative example of inter prediction, there is motion estimation coding. In motion estimation coding, blocks of a current image are predicted using a reference image. A reference block most similar to the current block may be searched for within a predetermined range by using a predetermined evaluation function. A current block is predicted based on a reference block, and a residual block is generated and encoded by subtracting a prediction block generated as a result of the prediction from the current block.

In order to derive a motion vector indicating a reference block in a reference picture, motion vectors of previously encoded blocks may be used as a motion vector predictor of a current block. A differential motion vector, which is a difference between the motion vector of the current block and the motion vector predictor, is signaled to the decoder side through a predetermined method.

Recently, technologies for encoding/decoding images using AI (Artificial Intelligent) have been proposed, and a method for effectively encoding/decoding images using AI, eg, a neural network, is required.

An image decoding method using AI according to an embodiment may include acquiring feature data of a current optical flow and feature data of a current residual image from a bitstream.

The image decoding method may include obtaining the current optical flow and first weight data by applying feature data of the current optical flow to an optical flow decoder.

The image decoding method may include obtaining the current residual image by applying feature data of the current residual image to a residual decoder.

The image decoding method may include obtaining a preliminary prediction image from a previous reconstructed image based on the current optical flow.

The image decoding method may include obtaining a final prediction image by applying sample values of the first weight data to sample values of the preliminary prediction image.

The image decoding method may include acquiring a current reconstructed image corresponding to the current image by combining the final prediction image and the current residual image.

An image decoding apparatus using AI according to an embodiment may include an acquisition unit that obtains feature data of a current optical flow and feature data of a current residual image from a bitstream.

The image decoding apparatus acquires the current optical flow and first weight data by applying feature data of the current optical flow to an optical flow decoder, and applies feature data of the current residual image to a residual decoder to obtain the current residual Obtaining an image, obtaining a preliminary prediction image from a previous reconstructed image based on the current optical flow, obtaining a final prediction image by applying sample values of the first weight data to sample values of the preliminary prediction image, A predictive decoding unit may be configured to obtain a current reconstructed image corresponding to the current image by combining the final predicted image and the current residual image.

An image encoding method using AI according to an embodiment may include obtaining characteristic data of a current optical flow by applying a current image and a previous reconstructed image to an optical flow encoder.

The image encoding method may include obtaining the current optical flow and first weight data by applying feature data of the current optical flow to an optical flow decoder.

The image encoding method may include obtaining a preliminary prediction image from a previous reconstructed image based on the current optical flow.

The image encoding method may include obtaining a final prediction image by applying sample values of the first weight data to sample values of the preliminary prediction image.

The image encoding method may include obtaining a current residual image corresponding to a difference between the final prediction image and the current image.

The image encoding method may include obtaining feature data of the current residual image by applying the current residual image to a residual encoder.

The image encoding method may include generating a bitstream including feature data of the current optical flow and feature data of the current residual image.

An image encoding apparatus using AI according to an embodiment obtains feature data of a current optical flow by applying a current image and a previous reconstructed image to an optical flow encoder, and applies the feature data of the current optical flow to an optical flow decoder to obtain Obtaining the current optical flow and first weight data, obtaining a preliminary prediction image from a previous reconstructed image based on the current optical flow, and applying sample values of the first weight data to sample values of the preliminary prediction image Prediction of obtaining a final predicted image, obtaining a current residual image corresponding to a difference between the final predicted image and the current image, and obtaining feature data of the current residual image by applying the current residual image to a residual encoder It may include an encoding unit.

The image encoding apparatus may include a generator configured to generate a bitstream including feature data of the current optical flow and feature data of the current residual image.

An image decoding method using AI according to an embodiment may include acquiring feature data of a current optical flow and feature data of a current residual image from a bitstream.

The image decoding method may include obtaining a plurality of current optical flows and a plurality of second weight data by applying characteristic data of the current optical flow to an optical flow decoder.

The image decoding method may include obtaining the current residual image by applying feature data of the current residual image to a residual decoder.

The image decoding method may include acquiring a plurality of preliminary predicted images from a previous reconstructed image based on the plurality of current optical flows.

The image decoding method may obtain a plurality of modified prediction images by applying sample values of the plurality of second weight data to sample values of the plurality of preliminary prediction images.

The image decoding method may include obtaining a final predicted image by combining the plurality of modified predicted images.

The image decoding method may include acquiring a current reconstructed image corresponding to the current image by combining the final prediction image and the current residual image.

An image encoding method using AI according to an embodiment may include obtaining characteristic data of a current optical flow by applying a current image and a previous reconstructed image to an optical flow encoder.

The image encoding method may include obtaining a plurality of current optical flows and a plurality of second weight data by applying feature data of the current optical flow to an optical flow decoder.

The image encoding method may include acquiring a plurality of preliminary prediction images from a previous reconstructed image based on the plurality of current optical flows.

The image encoding method may include obtaining a plurality of modified prediction images by applying sample values of the plurality of second weight data to sample values of the plurality of preliminary prediction images.

The image encoding method may include obtaining a final predicted image by combining the plurality of transformed predicted images.

The image encoding method may include obtaining a current residual image corresponding to a difference between the final predicted image and the current image.

The image encoding method may include obtaining feature data of the current residual image by applying the current residual image to a residual encoder.

The image encoding method may include generating a bitstream including feature data of the current optical flow and feature data of the current residual image.

1 is a diagram illustrating a process of encoding and decoding an image based on AI.
2 is a diagram illustrating a previous reconstruction image and a current image.
3 is a diagram illustrating a previous reconstruction image and a current image.
4 is a diagram for explaining an image encoding and decoding process according to an exemplary embodiment.
5 is a diagram for explaining an image encoding and decoding process according to an exemplary embodiment.
6 is a diagram illustrating a configuration of a video decoding apparatus according to an exemplary embodiment.
7 is a diagram illustrating a configuration of an acquisition unit according to an exemplary embodiment.
8 is a diagram showing the configuration of a predictive decoding unit according to an embodiment.
9 is a diagram showing the configuration of a prediction decoding unit according to an embodiment.
10 is a diagram showing the configuration of a prediction decoding unit according to an embodiment.
11 is a diagram showing the configuration of a predictive decoding unit according to an embodiment.
12 is a diagram showing the configuration of a prediction decoding unit according to an embodiment.
13 is a flowchart of an image decoding method according to an embodiment.
14 is a flowchart of an image decoding method according to an embodiment.
15 is a flowchart of an image decoding method according to an embodiment.
16 is a diagram illustrating a configuration of an image encoding apparatus according to an exemplary embodiment.
17 is a diagram showing the configuration of a predictive encoder according to an embodiment.
18 is a diagram showing the configuration of a predictive encoder according to an embodiment.
19 is a diagram illustrating a configuration of a generation unit according to an embodiment.
20 is a flowchart of an image encoding method according to an embodiment.
21 is a flowchart of an image encoding method according to an embodiment.
22 is a flowchart of an image encoding method according to an embodiment.
23 is a diagram illustrating the structure of a neural network according to an embodiment.
24 is a diagram for explaining a convolution operation in a convolution layer according to an embodiment.
25 is a diagram for explaining a training method of an optical flow encoder, an optical flow decoder, a residual encoder, and a residual decoder according to an embodiment.
26 is a diagram for explaining a training process of an optical flow encoder, an optical flow decoder, a residual encoder, and a residual decoder by a training apparatus according to an exemplary embodiment.

Since the present disclosure can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and will be described through detailed description. However, this is not intended to limit the embodiments of the present disclosure, and it should be understood that the present disclosure includes all modifications, equivalents, and substitutes included in the spirit and scope of the various embodiments.

In describing the embodiments, if it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter of the present disclosure, the detailed description will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the present disclosure are only identification symbols for distinguishing one component from another component.

In this disclosure, the expression “at least one of a, b, or c” means “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b” and c”, or variations thereof.

In addition, in the present disclosure, when one component is referred to as "connected" or "connected" to another component, the one component may be directly connected or directly connected to the other component, but in particular Unless otherwise described, it should be understood that they may be connected or connected via another component in the middle.

In addition, in the present disclosure, components expressed as '~ unit (unit)', 'module', etc., are two or more components combined into one component, or one component is divided into two or more for each function. may be differentiated into In addition, each of the components to be described below may additionally perform some or all of the functions of other components in addition to its own main function, and some of the main functions of each component may be different from other components. Of course, it may be performed exclusively by a component.

In addition, in the present disclosure, an 'image' may indicate a still image, a picture, a frame, a motion picture composed of a plurality of continuous still images, or a video.

In addition, in the present disclosure, a 'neural network' is a representative example of an artificial neural network model that mimics a cranial nerve, and is not limited to an artificial neural network model using a specific algorithm. A neural network may also be referred to as a deep neural network.

Also, in the present disclosure, a 'parameter' is a value used in an operation process of each layer constituting a neural network, and may be used, for example, when an input value is applied to a predetermined operation expression. A parameter is a value set as a result of training and can be updated through separate training data as needed.

Also, in the present disclosure, 'feature data' refers to data obtained by a neural network-based encoder processing input data. The feature data may be one-dimensional or two-dimensional data including several samples. Feature data may be referred to as a latent representation. The feature data may indicate features latent in data output by a decoder described later.

In addition, in the present disclosure, 'current image' means an image that is a current processing target, and 'previous image' means an image that is a processing target before the current image. Also, 'current optical flow' and 'current residual image' refer to optical flow and residual images obtained for processing the current image, respectively.

In addition, in the present disclosure, 'sample' refers to data to be processed as data allocated to a sampling position in an image, feature map, feature data, or weight data. For example, a sample may include a pixel in a 2D image.

1 is a diagram illustrating a process of encoding and decoding an image based on AI.

1 illustrates an inter prediction process. In inter prediction, an optical flow encoder 110, a residual encoder 130, an optical flow decoder 150, and a residual decoder 170 may be used.

The optical flow encoder 110, the residual encoder 130, the optical flow decoder 150, and the residual decoder 170 may be implemented as a neural network.

The optical flow encoder 110 and the optical flow decoder 150 may be understood as neural networks for extracting the current optical flow g from the current image 10 and the previous reconstructed image 30 .

The residual encoder 130 and the residual decoder 170 may be understood as neural networks for encoding and decoding the residual image r.

As described above, inter prediction is a process of encoding and decoding the current image 10 by using temporal redundancy between the current image 10 and the previous reconstruction image 30 . The previous reconstructed image 30 may be an image obtained by decoding a previous image that was a processing target before processing the current image 10 .

Position differences (or motion vectors) between blocks or samples in the current image 10 and reference blocks or reference samples in the previous reconstruction image 30 are used for encoding and decoding of the current image 10 . This positional difference may be referred to as optical flow. An optical flow may be defined as a set of motion vectors corresponding to samples or blocks in an image.

The current optical flow (g) indicates how the positions of samples in the previous reconstructed image 30 are changed within the current image 10, or whether samples identical/similar to those of the current image 10 are included in the previous reconstructed image 30. I can indicate where I am.

For example, if the same or most similar sample as the sample located at (1, 1) in the current image 10 is located at (2, 1) in the previous reconstruction image 30, the optical flow or motion vector for the sample can be derived as (1(=2-1), 0(=1-1)).

In the process of encoding and decoding an image using AI, the optical flow encoder 110 and the optical flow decoder 150 may be used to obtain the current optical flow g for the current image 10 .

Specifically, the previous reconstructed image 30 and the current image 10 may be input to the optical flow encoder 110 . The optical flow encoder 110 may output feature data w of the current optical flow by processing the current image 10 and the previous reconstructed image 30 according to parameters set as a result of training.

Characteristic data w of the current optical flow may be input to the optical flow decoder 150 . The optical flow decoder 150 may output the current optical flow g by processing the input feature data w according to parameters set as a training result.

The previous reconstructed image 30 is warped through warping 190 based on the current optical flow g, and as a result of the warping 190, a current prediction image x' may be obtained. Warping 190 is a type of geometric transformation that moves the positions of samples in an image.

Warping 190 is applied to the previous reconstructed image 30 according to the current optical flow (g) representing the relative positional relationship between the samples in the previous reconstructed image 30 and the samples in the current image 10, resulting in a current image A current prediction image (x') similar to (10) can be obtained.

For example, if the sample located at (1, 1) in the previous reconstruction image 30 is most similar to the sample located at (2, 1) in the current image 10, the previous reconstruction image ( 30), the position of the sample located at (1, 1) can be changed to (2, 1).

Since the current predicted image (x') generated from the previous reconstructed image (30) is not the current image (10) itself, the current residual image (r) between the current predicted image (x') and the current image (10) is obtained. It can be.

For example, the current residual image r may be obtained by subtracting sample values in the current predicted image (x′) from sample values in the current image 10 .

The current residual image r may be input to the residual encoder 130 . The residual encoder 130 may process the current residual image r according to parameters set as a result of training and output feature data v of the current residual image.

Feature data (v) of the current residual image may be input to the residual decoder 170 . The residual decoder 170 may output a restored current residual image r′ by processing the input feature data v according to a parameter set as a result of training.

A current reconstructed image 50 may be obtained by combining the current predicted image (x') and the reconstructed current residual image (r').

When the encoding and decoding processes of the image shown in FIG. 1 are implemented by an encoding device and a decoding device, the encoding device converts the characteristic data w of the current optical flow and the current residual image through the encoding process of the current image 10. Feature data (v) may be acquired. Then, the encoding device applies transform and/or quantization to feature data (w) of the current optical flow and feature data (v) of the current residual image, and generates a bitstream including a result of the transform and/or quantization, It can be transmitted to the decryption device. The decoding apparatus may obtain feature data (w) of the current optical flow and feature data (v) of the current residual image by applying inverse quantization and/or inverse transformation to data extracted from the bitstream. In addition, the decoding apparatus may process the feature data (w) of the current optical flow and the feature data (v) of the current residual image by the optical flow decoder 150 and the residual decoder 170 to obtain the current reconstructed image 50. there is.

As described above, the current residual image r between the current image 10 and the current predicted image x' is input to the residual encoder 130, and the feature data of the current residual image ( v) can be obtained. Since the feature data (v) of the current residual image is included in the bitstream after a predetermined process, in order to reduce the size of the bitstream, the size of the feature data (v) of the current residual image, furthermore, the current residual image (r) It is necessary to reduce the size of the sample values of . However, if the current optical flow (g) is inaccurate, the difference between the current image 10 and the current predicted image (x') inevitably increases, so the inaccurate current optical flow (g) increases the size of the bitstream become a factor

The current optical flow (g) may be obtained inaccurately due to limitations in the processing capabilities of the optical flow encoder 110 and the optical flow decoder 150, but in some cases, the previous reconstruction image 30 and the current image 10 Accuracy may decrease due to movement of included objects, and accordingly, the quality of the current predicted image (x') may also decrease.

A case in which a current optical flow g of low quality can be obtained will be described with reference to FIGS. 2 and 3 .

2 is a diagram illustrating a previous reconstruction image 30 and a current image 10 .

Movements of objects included in consecutive images may vary. Here, an object may mean a sample or a set of samples.

For example, the movement of one object included in the continuous images may be large, and the movement of another object may be small. Also, for example, an object included in any one image may not exist in the previous or next image.

A large movement of a predetermined object means that the positional difference of the predetermined object within consecutive images is large, and a small movement of a predetermined object means that the positional difference of the predetermined object within consecutive images is small. can do.

The current optical flow g obtained from the current image 10 and the previous reconstructed image 30 may become inaccurate due to various motions and occlusion of objects included in the images.

Referring to FIG. 2 , if the difference between the position of an object included in area A-1 in the current image 10 and the position of the same object included in area A-2 in the previous reconstructed image 30 is not large, the corresponding object movement can be considered small. For an object with small motion, the current optical flow (g) can be measured relatively accurately. This is because only the periphery of the area corresponding to area A-1 in the previous reconstructed image 30 needs to be searched to obtain the motion vector of the object included in area A-1 in the current image 10 .

In addition, if the difference between the position of an object included in area B-1 in the current image 10 and the position of the same object included in area B-2 in the previous reconstructed image 30 is large, the movement of the corresponding object is considered to be large. can do. Currently, it is difficult to accurately measure the optical flow (g) for a moving object. Because, in order to obtain a motion vector of an object included in area B-1 in the current image 10, even if the area corresponding to area B-1 is searched in the previous reconstructed image 30, it is included in area B-1. This is because it is difficult to find the same object (ie, an object included in the B-2 area).

In addition, the object included in region C in the current image 10 may not be included in the previous reconstruction image 30 . In other words, an object included in region C may be occluded in the previous reconstruction image 30 . In this case, it is difficult to find an object identical/similar to an object included in region C in the current image 10 in the previous reconstruction image 30 .

In the case of an object with small motion, since a motion vector can be calculated relatively accurately, encoding/decoding efficiency can be increased when inter prediction is applied to the object. Here, high encoding/decoding efficiency means that the bitrate of a bitstream is reduced and the quality of a reconstructed object is increased.

Conversely, in the case of an object with large motion or an occluded object, since it is difficult to accurately calculate a motion vector, encoding/decoding efficiency is reduced when inter prediction is applied to the corresponding object.

On the other hand, even if there is little movement of objects included in the current image 10 or objects included in the current image 10 are not occluded in the previous reconstructed image 30, it is not appropriate to apply inter prediction. There may be cases.

Referring to FIG. 3 , even if there is no or very small difference in position between an object included in area R-1 in the current image 10 and the same object included in area R-2 in the previous reconstruction image 30, the object is located in area R-1. If the difference between the brightness of an included object and the brightness of an object included in the R-2 region in the previous reconstructed image 30 is large, it may not be appropriate to apply inter prediction to the object.

As described with reference to FIG. 1, the current residual image (r) corresponding to the difference between the current predicted image (x') and the current image (10) generated through the warping (190) of the previous reconstructed image (30) This residual may be input to the encoder 130. At this time, if the difference between the brightness value of a specific object in the current image 10 and the brightness value of the same object in the previous reconstructed image 30 is large, the sample values in the current image 10 and the previous reconstructed image 30 generate A difference between sample values at the same position in the current prediction image (x') also increases, and as a result, the sample value of the current residual image (r) increases. That is, if the change in the brightness value of an object in consecutive images is large, it may be difficult to increase encoding/decoding efficiency even if inter prediction is applied to the corresponding object.

As described above, encoding/decoding efficiency may increase when inter prediction is applied to an object included in consecutive images, and encoding/decoding efficiency may decrease when inter prediction is applied to an object.

Therefore, rather than using the current predicted image (x') as it is, encoding/decoding efficiency can be improved by transforming the current predicted image (x') according to the characteristics of the object and using the modified predicted image.

4 is a diagram for explaining an image encoding and decoding process according to an exemplary embodiment.

Referring to FIG. 4 , an optical flow encoder 410 , an optical flow decoder 450 , a residual encoder 430 , and a residual decoder 470 may be used to encode and decode an image.

In one embodiment, the optical flow encoder 410, the optical flow decoder 450, the residual encoder 430, and the residual decoder 470 may be implemented as a neural network.

For encoding of the current image 420 , the previous reconstructed image 440 and the current image 420 may be input to the optical flow encoder 410 . The optical flow encoder 410 may output feature data w of the current optical flow by processing the current image 420 and the previous reconstructed image 440 according to parameters set as a result of training.

Characteristic data w of the current optical flow may be input to the optical flow decoder 450 .

The optical flow decoder 450 may output the current optical flow g by processing the input feature data w according to parameters set as a training result.

The optical flow decoder 450 may output first weight data t1 separately from the current optical flow g. The first weight data t1 is used to transform a preliminary prediction image (x') generated through warping 490 .

The previous reconstructed image 440 is warped through warping 490 based on the current optical flow g, and as a result of the warping 490, a preliminary prediction image x' may be obtained.

A final prediction image (x″) may be obtained by applying the first weight data (t1) to the preliminary prediction image (x′).

As will be described later with reference to FIGS. 25 and 26, the optical flow decoder 450 reduces the bit rate of the bitstream including the feature data v of the current residual image or the sample values of the current residual image r. Since training can be performed, the first weight data t1 that reduces the difference between the current image 420 and the preliminary prediction image x' can be output from the optical flow decoder 450 .

A current residual image (r) corresponding to a difference between the final predicted image (x″) and the current image 420 may be input to the residual encoder 430 .

The residual encoder 430 may process the current residual image r according to parameters set as a result of training and output feature data v of the current residual image.

Feature data v of the current residual image may be input to the residual decoder 470 . The residual decoder 470 may obtain a reconstructed current residual image r' by processing the feature data v according to parameters set as a result of training.

A current reconstructed image 460 may be obtained by combining the reconstructed current residual image (r') and the final predicted image (x'').

When the process of encoding and decoding the image shown in FIG. 4 is implemented by an encoding device and a decoding device, the encoding device converts the feature data w of the current optical flow and the characteristics of the current residual image through encoding of the current image 420. Data (v) can be obtained. Also, the encoding device may generate a bitstream including feature data (w) of the current optical flow and feature data (v) of the current residual image, and transmit the generated bitstream to the decoding device.

The decoding apparatus may obtain feature data (w) of the current optical flow and feature data (v) of the current residual image from the bitstream. Also, the decoding apparatus may obtain a current reconstructed image 460 based on feature data (w) of the current optical flow and feature data (v) of the current residual image.

In the process of encoding and decoding the image shown in FIG. 4, the process shown in FIG. 1 is different in that the preliminary prediction image (x') generated through warping 490 is transformed according to the first weight data t1. .

That is, by processing the preliminary predicted image (x') based on the first weight data (t1), the final predicted image (x'') can be similar to the current image 420, thereby improving encoding/decoding efficiency. It can be.

Specifically, the sample value of the preliminary prediction image (x') may be changed to a small value by outputting the first weight data (t1) including a small sample value for an object with large motion or an occluded object, and the motion may be reduced. For a small or non-moving object, the first weight data t1 including a sample value close to 1 is output, so that the sample value of the preliminary prediction image x' may hardly change.

In other words, the result of inter prediction according to the sample values included in the first weight data t1, that is, the influence of the sample values of the preliminary prediction image (x') on the sample values of the final prediction image (x'') This may vary from sample to sample.

A sample value of 0 included in the first weight data t1 means that the sample value of the preliminary prediction image (x') has no effect on the sample value of the final prediction image (x''), which means that the corresponding It can be understood that inter prediction is not applied to the sample. In this case, a sample of the current reconstructed image 460 is obtained, and since the previous reconstructed image 440 is not used, it can be understood that intra prediction is applied.

In one embodiment, the sample value included in the first weight data t1 is set to be greater than 1 or greater than 1 for an object in the previous reconstruction image 440 whose brightness value change is greater than that of the current image 420. By setting it small, the sample value of the preliminary prediction image (x') can be made more similar to the sample value in the current image 420 .

Meanwhile, the number of samples included in the current optical flow (g) may be the same as the number of samples included in the current image 420, which is one motion vector for each sample included in the current image 420 means that is extracted. However, since several samples identical/similar to the samples included in the current image 420 may exist in the previous reconstruction image 440, two or more motion vectors are extracted for each of the samples of the current image 420. In this case, the efficiency of inter prediction may be better. In this regard, it will be described with reference to FIG. 5 .

Referring to FIG. 5 , an optical flow encoder 410 , an optical flow decoder 450 , a residual encoder 430 , and a residual decoder 470 may be used to encode and decode an image.

In one embodiment, the optical flow encoder 410, the optical flow decoder 450, the residual encoder 430, and the residual decoder 470 may be implemented as a neural network.

For encoding of the current image 420 , the previous reconstructed image 440 and the current image 420 may be input to the optical flow encoder 410 .

The optical flow encoder 410 may output feature data w of the current optical flow by processing the current image 420 and the previous reconstructed image 440 according to parameters set as a result of training.

Characteristic data w of the current optical flow may be input to the optical flow decoder 450 .

The optical flow decoder 450 may output a plurality of current optical flows g[i] (i being an index and an integer greater than or equal to 0) by processing the input feature data w according to parameters set as a result of training. there is.

The optical flow decoder 450 may output a plurality of second weight data t2[i] separately from the plurality of current optical flows g[i]. The plurality of second weight data t2[i] transforms the plurality of preliminary prediction images x′[i] generated through warping 490-1, 490-2, ..., 490-n. can be used

The plurality of second weight data t2[i] may be referred to as reliability or accuracy of the plurality of current optical flows g[i].

The number of samples included in each of the plurality of second weight data t2[i] may be the same as the number of samples included in each of the plurality of current optical flows g[i]. In this case, the plurality of Sample values included in the second weight data t2[i] may indicate reliability or accuracy of sample values included in the plurality of current optical flows g[i]. For example, sample values at predetermined positions in the plurality of second weight data t2[i] may indicate reliability or accuracy of sample values at predetermined positions in the plurality of current optical flows g[i]. . As will be described later, by setting the sum of sample values at the same position in the plurality of second weight data t2[i] to 1, the sample values of the previous reconstructed image 440 form the final predicted image (x''). You can also keep it as it is.

The previous reconstruction image 440 is warped through warping (490-1, 490-2, ..., 490-n) based on a plurality of current optical flows (g[i]), and warping (490- As a result of 1, 490-2, ..., 490-n), a plurality of preliminary prediction images x'[i] may be obtained.

A plurality of preliminary prediction images (t2[i]) are applied to a plurality of preliminary prediction images (x'[i]), and a plurality of preliminary prediction images (t2[i]) are applied. By combining x'[i]), a final predicted image (x'') may be obtained.

Since the optical flow decoder 450 can be trained to reduce the bitrate of the bitstream including the feature data v of the current residual image or the sample values of the current residual image r, the current image 420 and final prediction The optical flow decoder 450 may output a plurality of second weight data t2[i] to reduce the difference between the images x''.

A current residual image (r) corresponding to a difference between the final prediction image (x″) and the current image 420 may be input to the residual encoder 430 .

The residual encoder 430 may process the current residual image r according to parameters set as a result of training and output feature data v of the current residual image.

Feature data v of the current residual image may be input to the residual decoder 470 . The residual decoder 470 may obtain a reconstructed current residual image r' by processing the feature data v according to parameters set as a result of training.

A current reconstructed image 460 may be obtained by combining the reconstructed current residual image (r') and the final predicted image (x'').

When the process of encoding and decoding the image shown in FIG. 5 is implemented by an encoding device and a decoding device, the encoding device uses the feature data w of the current optical flow obtained through encoding of the current image 420 and the current residual image A bitstream including feature data (v) of can be generated and transmitted to a decoding device.

The decoding apparatus may obtain feature data (w) of the current optical flow and feature data (v) of the current residual image from the bitstream. Also, the decoding apparatus may obtain a current reconstructed image 460 based on feature data (w) of the current optical flow and feature data (v) of the current residual image.

In the process of encoding and decoding the image shown in FIG. 5, the optical flow decoder 450 outputs a plurality of current optical flows g[i] and a plurality of second weight data t2[i]. and different from the process shown in FIG. 4 .

That is, a plurality of preliminary prediction images that can function as a predicted version of the current image 420 are obtained based on a plurality of current optical flows (g[i]), and a plurality of current optical flows (g[i] ]) is applied to the plurality of preliminary prediction images, thereby improving encoding/decoding efficiency.

Hereinafter, with reference to FIGS. 6 to 22 , an image decoding apparatus 600 , an image encoding apparatus 1600 , and a method for decoding/encoding an image using the video decoding apparatus 600 according to an exemplary embodiment will be described.

6 is a diagram illustrating a configuration of an image decoding apparatus 600 according to an exemplary embodiment.

Referring to FIG. 6 , an image decoding apparatus 600 according to an embodiment may include an acquisition unit 610 and a prediction decoding unit 630.

The acquisition unit 610 and the prediction decoding unit 630 may be implemented with at least one processor. The acquisition unit 610 and the prediction decoding unit 630 may operate according to at least one instruction stored in a memory (not shown).

Although FIG. 6 shows the acquisition unit 610 and the prediction decoding unit 630 separately, the acquisition unit 610 and the prediction decoding unit 630 may be implemented by a single processor. In this case, the acquisition unit 610 and the prediction decoding unit 630 may be implemented as a dedicated processor, and a general-purpose processor such as an AP (application processor), CPU (central processing unit) or GPU (graphic processing unit) and software It can also be implemented through a combination. In addition, a dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processing unit for using an external memory.

The acquisition unit 610 and the prediction decoding unit 630 may be composed of a plurality of processors. In this case, the acquisition unit 610 and the prediction decoding unit 630 may be implemented by a combination of dedicated processors, or may be implemented by a combination of software and a plurality of general-purpose processors such as APs, CPUs, or GPUs.

The acquisition unit 610 may obtain a bitstream including an encoding result of the current image.

The acquisition unit 610 may receive a bitstream from an image encoding device through a network. In one embodiment, the acquisition unit 610 may be a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, and a magneto-optical medium such as a floptical disk. -optical medium) and the like may obtain a bitstream from a data storage medium.

The acquisition unit 610 may acquire feature data of the current optical flow and feature data of the current residual image by parsing the bitstream.

In an embodiment, the acquisition unit 610 acquires a first bitstream corresponding to feature data of a current optical flow and a second bitstream corresponding to feature data of a current residual image, and obtains the first bitstream and the second bitstream Characteristic data of the current optical flow and feature data of the current residual image may be obtained by parsing each stream.

The feature data of the current optical flow and the feature data of the current residual image are transferred to the predictive decoding unit 630, and the predictive decoding unit 630 uses the feature data of the current optical flow and the feature data of the current residual image to obtain a current image. A corresponding current reconstructed image may be acquired. The current reconstruction image may be output to a display device for reproduction.

7 is a diagram illustrating a configuration of an acquisition unit 610 according to an exemplary embodiment.

Referring to FIG. 7 , an acquisition unit 610 may include an entropy decoding unit 611 and an inverse quantization unit 613 .

The entropy decoding unit 611 may entropy code bins included in the bitstream to obtain quantized feature data of the current optical flow and quantized feature data of the current residual image.

The inverse quantization unit 613 obtains feature data of the current optical flow by inverse quantizing the quantized feature data of the current optical flow, and inverse quantizes the quantized feature data of the current residual image to obtain feature data of the current residual image. can

In one embodiment, the acquisition unit 610 may further include an inverse transform unit. The inverse transform unit may inverse transform the feature data output from the inverse quantization unit 613 from the frequency domain to the spatial domain. When the image encoding apparatus 1600 described later converts the feature data of the current optical flow and the feature data of the current residual image from the spatial domain to the frequency domain, the inverse transform unit converts the feature data of the current optical flow output from the inverse quantization unit 613 and feature data of the current residual image may be inversely transformed from the frequency domain to the spatial domain.

In one embodiment, the acquisition unit 610 may not include the inverse quantization unit 613. That is, feature data of the current optical flow and feature data of the current residual image may be obtained through processing by the entropy decoding unit 611 .

In an embodiment, the acquisition unit 610 may acquire feature data of the current optical flow and feature data of the current residual image by performing only inverse binarization on bins included in the bitstream. This is when the image encoding apparatus 1600 generates a bitstream by binarizing the feature data of the current optical flow and the feature data of the current residual image. This is for a case where entropy encoding, transformation, and quantization are not applied to the feature data of the residual image.

8 is a diagram showing the configuration of a prediction decoding unit 630 according to an embodiment.

Referring to FIG. 8 , the prediction decoding unit 630 may include an optical flow decoder 450, a residual decoder 470, a motion compensation unit 631, a weight application unit 632, and a combiner 633. .

The optical flow decoder 450 and the residual decoder 470 may be implemented as a neural network including at least one layer (eg, a convolutional layer).

The optical flow decoder 450 and the residual decoder 470 may be stored in memory. In one embodiment, the optical flow decoder 450 and the residual decoder 470 may be implemented with at least one dedicated processor for AI.

Feature data of the current optical flow and feature data of the current residual image output by the acquisition unit 610 may be input to the optical flow decoder 450 and the residual decoder 470, respectively.

The optical flow decoder 450 may process feature data of the current optical flow according to parameters set through training and output the current optical flow and first weight data.

The current optical flow and first weight data are 1-dimensional or 2-dimensional data and may consist of a plurality of samples. The size of the current optical flow and the size of the first weight data may be the same. Here, the same size may mean that the number of samples is the same.

In one embodiment, the size of the current optical flow and the size of the first weight data may be the same as the size of the previous reconstructed image.

The current optical flow may be provided to the motion compensator 631 . The motion compensator 631 may generate a preliminary prediction image by processing the previous reconstructed image according to the current optical flow.

The motion compensator 631 may warp a previous reconstructed image according to the current optical flow to generate a preliminary prediction image. Warping for generating a preliminary prediction image is an example, and the motion compensator 631 may perform various image processing to change a previous reconstructed image based on a current optical flow to generate a preliminary prediction image.

The preliminary prediction image generated by the motion compensator 631 and the first weight data output by the optical flow decoder 450 may be provided to the weight application unit 632 .

The weight application unit 632 may obtain a final prediction image by applying sample values of the first weight data to sample values of the preliminary prediction image.

In an embodiment, the weight application unit 632 may obtain a final prediction image by multiplying sample values of the first weight data by sample values of the preliminary prediction image. In this case, the weight application unit 632 may be referred to as a multiplier.

For example, the first weight data includes a first sample value and a second sample value, the preliminary prediction image has a first sample value at the same position as the first sample value of the first weight data, and the first weight data If it includes the second sample value at the same position as the second sample value of the weight data, the first sample value of the final prediction image is obtained by multiplying the first sample value of the first weight data by the first sample value of the preliminary prediction image. , and a second sample value of the final prediction image may be obtained by multiplying the second sample value of the first weight data by the second sample value of the preliminary prediction image.

The first weight data may include weights for making sample values of the final prediction image as similar as possible to sample values of the current image. As described later with reference to FIGS. 25 and 26 , the optical flow decoder 450 may be trained in a direction in which sample values of the current residual image corresponding to the difference between the current image and the final predicted image are reduced or minimized. Accordingly, the optical flow decoder 450 may output first weight data for reducing the difference between the current image and the final predicted image.

In an embodiment, the sample value in the first weight data may have a value closer to 1 as the difference between the sample value in the preliminary prediction image and the sample value in the same position in the current image is smaller. In this case, it can be understood that the application rate of inter prediction to the corresponding sample increases.

Also, in an embodiment, when a sample value in the preliminary prediction image is greater than a sample value at the same position in the current image, the sample value in the first weight data may have a value smaller than 1. Also, in one embodiment, when a sample value in the preliminary prediction image is smaller than a sample value at the same location in the current image, the sample value in the first weight data may have a value greater than 1.

The first weight data may be for compensating for various motions and brightness changes of objects included in the current image and the previous reconstruction image.

As described above, if the movement of the object included in the current image and the previous reconstructed image is large, the sample value of the current optical flow may be inaccurate. In this case, if the sample value of the preliminary prediction image is not changed, the current residual image The higher the sample value, the higher the probability.

The optical flow decoder 450 may output first weight data for reducing an application rate of inter prediction to the corresponding object when the movement of a predetermined object included in the current image and the previous reconstructed image is large. As the sample value included in the first weight data decreases, the application rate of inter prediction may also decrease. If the sample value included in the first weight data is 0, the application rate of inter prediction may be 0%.

In addition, the optical flow decoder 450 may output first weight data for increasing the inter-prediction application rate for a given object included in the current image and the previous reconstructed image if the motion of the predetermined object is small or there is no motion. there is. As the sample value included in the first weight data has a value close to 1, an application rate of inter prediction may increase.

In addition, the optical flow decoder 450 may output first weight data capable of compensating for the brightness change, if there is a change in brightness of a predetermined object included in the current image and the previous reconstructed image. For example, if the brightness of a predetermined object included in a previous reconstructed image is increased in the current image, the sample value in the first weight data may have a value greater than 1. Also, if the brightness of a predetermined object included in the previous reconstructed image is reduced in the current image, the sample value in the first weight data may have a value less than 1.

According to an embodiment, since the adjustment of the application rate of inter prediction and the compensation for the brightness change can be performed for each sample of the preliminary prediction image through the first weight data, sample values of the current residual image can be minimized, and thus the bit The bitrate of the stream may also be reduced.

The final predicted image obtained by the weight application unit 632 may be provided to the combiner 633.

Feature data of the current residual image output by the acquisition unit 610 may be input to the residual decoder 470 .

In an embodiment, feature data of the current optical flow and feature data of the current residual image may be concatenated, and the concatenated result may be input to the residual decoder 470 . Here, concatenation may mean a process of combining two or more feature data in a channel direction.

The residual decoder 470 may obtain a current residual image by processing feature data of the current residual image according to parameters set through training. The current residual image may be provided to the combiner 633.

The combiner 633 may obtain a current reconstructed image by combining the final prediction image and the current residual image.

In an embodiment, the combiner 633 may obtain a current reconstructed image by adding sample values of the final prediction image and sample values of the current residual image. In this case, the combiner 633 may be referred to as an adder.

9 is a diagram showing the configuration of a predictive decoding unit 630-1 according to an embodiment.

The prediction decoding unit 630-1 shown in FIG. 9 is the same as the prediction decoding unit 630 shown in FIG. 8 except that the first weight data is output from the weight decoder 634.

Specifically, the weight decoder 634 may be implemented as a neural network including at least one layer (eg, a convolutional layer).

Weight decoder 634 may be stored in memory. In one embodiment, weight decoder 634 may be implemented with at least one dedicated processor for AI.

Characteristic data of the current optical flow output by the acquisition unit 610 may be input to the optical flow decoder 450 and the weight decoder 634 .

The optical flow decoder 450 may output the current optical flow by processing feature data of the current optical flow according to parameters set through training.

The weight decoder 634 may output first weight data by processing feature data of the current optical flow according to parameters set through training.

Like the optical flow decoder 450 shown in FIG. 8, the weight decoder 634 shown in FIG. 9 is also trained in a direction in which sample values of the current residual image corresponding to the difference between the current image and the final predicted image are reduced or minimized. It can be. Accordingly, the weight decoder 634 may output first weight data for reducing the difference between the current image and the final predicted image.

In an embodiment, the sample value in the first weight data may have a value closer to 1 as the difference between the sample value in the preliminary prediction image and the sample value in the same position in the current image is smaller. In this case, it can be understood that the application rate of inter prediction to the corresponding sample increases.

Also, in an embodiment, when a sample value in the preliminary prediction image is greater than a sample value at the same position in the current image, the sample value in the first weight data may have a value smaller than 1. Also, in one embodiment, when a sample value in the preliminary prediction image is smaller than a sample value at the same location in the current image, the sample value in the first weight data may have a value greater than 1.

The first weight data may be understood as compensation for various motions and brightness changes of objects included in the current image and the previous reconstruction image.

10 is a diagram showing the configuration of a prediction decoding unit 630-2 according to an embodiment.

In the prediction decoding unit 630-2 shown in FIG. 10, the prediction decoding unit 630-2 shown in FIG. 9 except that the weight decoder 634 processes feature data of the weight data and outputs first weight data. Same as 1).

Feature data of the weight data may be generated by the image encoding apparatus 1600 and included in the bitstream. The acquisition unit 610 may acquire feature data of the weight data from the bitstream and provide the obtained feature data to the weight decoder 634 .

Since other components of the prediction decoding unit 630-2 have been described with reference to FIGS. 8 and 9, a detailed description thereof will be omitted.

11 is a diagram showing the configuration of a prediction decoding unit 630-3 according to an embodiment.

The prediction decoding unit 630-3 shown in FIG. 11 may include an optical flow decoder 450, a residual decoder 470, a motion compensation unit 631, a weight application unit 632, and a combiner 633. there is.

The optical flow decoder 450 and the residual decoder 470 may be implemented as a neural network including at least one layer (eg, a convolutional layer).

The optical flow decoder 450 and the residual decoder 470 may be stored in memory. In one embodiment, the optical flow decoder 450 and the residual decoder 470 may be implemented with at least one dedicated processor for AI.

Feature data of the current optical flow and feature data of the current residual image output by the acquisition unit 610 may be input to the optical flow decoder 450 and the residual decoder 470, respectively.

The optical flow decoder 450 may output a plurality of current optical flows and a plurality of second weight data by processing feature data of a current optical flow according to parameters set through training.

The plurality of current optical flows and the plurality of second weight data are one-dimensional or two-dimensional data, and may consist of a plurality of samples. The size of each current optical flow and the size of each second weight data may be the same.

In an embodiment, the size of each current optical flow and the size of each second weight data may be the same as the size of a previous reconstructed image.

In one embodiment, a plurality of current optical flows and a plurality of second weight data may correspond 1:1. For example, if five current optical flows are output by the optical flow decoder 450, the number of second weight data output by the optical flow decoder 450 may also be five.

A plurality of current optical flows may be provided to the motion compensator 631 . The motion compensation unit 631 may generate a plurality of preliminary prediction images by processing previous reconstructed images according to a plurality of current optical flows.

The motion compensator 631 may warp previous reconstructed images according to each of a plurality of current optical flows in order to generate a plurality of preliminary prediction images. Warping for generating a plurality of preliminary prediction images is one example, and the motion compensator 631 performs various image processing to change previous reconstruction images based on a plurality of current optical flows in order to generate a plurality of preliminary prediction images. can be performed.

A plurality of preliminary prediction images generated by the motion compensator 631 and a plurality of second weight data output by the optical flow decoder 450 may be provided to the weight application unit 632 .

The plurality of second weight data may be referred to as reliability or accuracy of a plurality of current optical flows.

Sample values included in the plurality of second weight data may indicate reliability or accuracy of sample values included in the plurality of current optical flows. For example, sample values at predetermined positions in a plurality of second weight data may indicate reliability or accuracy of sample values at predetermined positions in a plurality of current optical flows. Also, for example, each of the sample values included in any one of the second weight data may represent the reliability or accuracy of each of the sample values included in the current optical flow corresponding to the one of the second weight data, and the other one Each of the sample values included in the second weight data of may indicate reliability or accuracy of each sample value included in the current optical flow corresponding to another second weight data.

The weight application unit 632 may obtain a plurality of transformed prediction images by applying a plurality of second weight data to a plurality of preliminary prediction images. For example, the weight application unit 632 multiplies sample values of any one of the second weight data by sample values of a preliminary prediction image corresponding to any one of the second weight data to obtain any one modified prediction image. can And, the weight application unit 632 may obtain another modified prediction image by multiplying sample values of another piece of second weight data by sample values of a preliminary prediction image corresponding to another piece of second weight data. .

As described above, since the plurality of second weight data may mean the reliability or accuracy of the plurality of current optical flows, the final prediction image more similar to the current image is generated by applying the reliability or accuracy to the plurality of preliminary prediction images. It can be.

In one embodiment, the sum of sample values at the same location within the plurality of second weight data may be 1. Accordingly, the positions of samples of the final predicted image may be different compared to those of the previous reconstructed image, but sample values of the previous reconstructed image may be maintained in the final predicted image.

A plurality of modified prediction images may be provided to the combiner 633 . The combiner 633 may obtain a final predicted image by combining a plurality of transformed predicted images. In an embodiment, the combining unit 633 may obtain a final predicted image by summing sample values of the plurality of transformed predicted images at the same location.

The residual decoder 470 may obtain a current residual image by processing feature data of the current residual image according to parameters set through training. The current residual image may be provided to the combiner 633.

The combiner 633 may obtain a current reconstructed image by combining the final prediction image and the current residual image.

In an embodiment, the combiner 633 may obtain a current reconstructed image by adding sample values of the final prediction image and sample values of the current residual image.

The prediction decoding unit 630-3 shown in FIG. 11 corresponds to the encoding/decoding process of an image previously described with respect to FIG. It can be understood as a positional difference between samples, that is, a set of motion vectors.

In other words, the predictive decoding unit 630-3 shown in FIG. 11 obtains a set of several motion vectors and uses second weight data representing their reliability or accuracy to obtain a final predicted image more similar to the current image. can be obtained

Meanwhile, although not shown in FIG. 11 , in one embodiment, the predictive decoding unit 630 may further include a weight decoder for outputting a plurality of second weight data by processing feature data of the current optical flow. In this case, the optical flow decoder 450 may output a plurality of current optical flows by processing feature data of the current optical flow, and output a plurality of second weight data by the weight decoder.

Also, in one embodiment, the weight decoder may output a plurality of second weight data by processing feature data of the weight data. Feature data of the weight data may be included in the bitstream, and the acquisition unit 610 may obtain feature data of the weight data from the bitstream and provide the obtained feature data to a weight decoder.

Also, in one embodiment, the prediction decoding unit 630 - 3 may include a plurality of optical flow decoders 450 outputting a pair of current optical flow and second weight data.

Also, in one embodiment, the prediction decoding unit 630 - 3 may include a plurality of optical flow decoders 450 and a plurality of weight decoders. In this case, each of the plurality of optical flow decoders 450 may output one (or more) current optical flows, and each of the plurality of weight decoders may output one (or more) second weight data.

12 is a diagram illustrating a configuration of a prediction decoding unit 630-4 according to an embodiment.

The prediction decoding unit 630-4 shown in FIG. 12 may be for performing the operations of the prediction decoding unit 630 shown in FIG. 8 and the prediction decoding unit 630-3 shown in FIG.

Referring to FIG. 12, the prediction decoding unit 630-4 may include an optical flow decoder 450, a residual decoder 470, a motion compensation unit 631, a weight application unit 632, and a combiner 633. can

The optical flow decoder 450 and the residual decoder 470 may be implemented as a neural network including at least one layer (eg, a convolutional layer).

The optical flow decoder 450 and the residual decoder 470 may be stored in memory. In one embodiment, the optical flow decoder 450 and the residual decoder 470 may be implemented with at least one dedicated processor for AI.

Feature data of the current optical flow and feature data of the current residual image output by the acquisition unit 610 may be input to the optical flow decoder 450 and the residual decoder 470, respectively.

The optical flow decoder 450 may process feature data of the current optical flow according to parameters set through training and output a plurality of current optical flows, first weight data, and a plurality of second weight data.

Each current optical flow, each of the first weight data and each of the second weight data is one-dimensional or two-dimensional data, and may consist of a plurality of samples. The size of each current optical flow, the size of the first weight data, and the size of each of the second weight data may be equal to each other.

In an embodiment, the size of each current optical flow, the size of each first weight data, and the size of each second weight data may be the same as the size of a previous reconstructed image.

In one embodiment, a plurality of current optical flows and a plurality of second weight data may correspond 1:1.

A plurality of current optical flows may be provided to the motion compensator 631 . The motion compensation unit 631 may generate a plurality of preliminary prediction images by processing previous reconstructed images according to a plurality of current optical flows.

The motion compensator 631 may warp previous reconstructed images according to each of a plurality of current optical flows in order to generate a plurality of preliminary prediction images. Warping for generating a plurality of preliminary prediction images is one example, and the motion compensator 631 performs various image processing to change previous reconstruction images based on a plurality of current optical flows in order to generate a plurality of preliminary prediction images. can be performed.

The plurality of preliminary prediction images generated by the motion compensator 631 and the first weight data and the plurality of second weight data output by the optical flow decoder 450 may be provided to the weight application unit 632 .

The plurality of second weight data may be referred to as reliability or accuracy of a plurality of current optical flows.

The weight application unit 632 may obtain a plurality of transformed prediction images by applying a plurality of second weight data to a plurality of preliminary prediction images. For example, the weight application unit 632 multiplies sample values of any one of the second weight data by sample values of a preliminary prediction image corresponding to any one of the second weight data to obtain any one modified prediction image. can And, the weight application unit 632 may obtain another modified prediction image by multiplying sample values of another piece of second weight data by sample values of a preliminary prediction image corresponding to another piece of second weight data. .

In one embodiment, the sum of sample values at the same location within the plurality of second weight data may be 1.

A plurality of modified predicted images may be provided to the combiner 633 . The combining unit 633 may obtain an intermediate prediction image by combining a plurality of modified prediction images.

In an embodiment, the combiner 633 may obtain an intermediate prediction image by summing sample values of the plurality of modified prediction images at the same location.

The intermediate prediction image may be provided to the weight application unit 632 . The weight application unit 632 may obtain a final prediction image by applying sample values of the first weight data to sample values of the intermediate prediction image.

In an embodiment, the weight application unit 632 may obtain a final prediction image by multiplying sample values of the intermediate prediction image by sample values of the first weight data.

The final predicted image obtained by the weight application unit 632 may be provided to the combiner 633.

The residual decoder 470 may obtain a current residual image by processing feature data of the current residual image according to parameters set through training. The current residual image may be provided to the combiner 633.

The combiner 633 may obtain a current reconstructed image by combining the final prediction image and the current residual image.

Although not shown in FIG. 12, in one embodiment, the prediction decoding unit 630-4 processes feature data of the current optical flow and outputs first weight data and a plurality of second weight data. At least one weight decoder and at least one optical flow decoder 450 outputting a plurality of current optical flows by processing feature data of the current optical flow.

In one embodiment, the at least one weight decoder includes a weight decoder for processing feature data of the current optical flow and outputting first weight data, and a weight decoder for outputting a plurality of second weight data by processing feature data of the current optical flow. It may also include a decoder.

Also, in one embodiment, at least one weight decoder may output first weight data and a plurality of second weight data by processing feature data of the weight data. In this case, feature data of the weight data may be included in the bitstream, and the acquisition unit 610 may obtain feature data of the weight data from the bitstream and provide the obtained feature data to the weight decoder.

In addition, in one embodiment, the prediction decoding unit 630-4 includes at least one optical flow decoder 450 outputting a plurality of current optical flows and first weight data and at least one outputting a plurality of second weight data. may include a weight decoder of

In addition, in one embodiment, the prediction decoding unit 630-4 includes at least one optical flow decoder 450 outputting a plurality of current optical flows and a plurality of second weight data and a weight decoder outputting first weight data. may include.

In addition, in one embodiment, the prediction decoding unit 630-4 includes a plurality of optical flow decoders 450 each outputting a pair of current optical flow and second weight data, and optical flow decoders 450 outputting first weight data. It may also include a flow decoder 450 (or weight decoder).

Meanwhile, the prediction decoding unit 630-4 shown in FIG. 12 obtains a plurality of transformed prediction images by multiplying a plurality of second weight data by a plurality of preliminary prediction images, and uses a combination result of the plurality of transformed prediction images. A final prediction image can be obtained by multiplying the corresponding intermediate prediction image by the first weight data, which can be expressed by Equation 1 below. In Equation 1 below, i represents an index.

[Equation 1]

Final prediction image = (∑(Preliminary prediction image[i]*Second weight data[i]))*First weight data

Equation 1 below may be modified as in Equation 2 below.

[Equation 2]

Final prediction image = ∑ (preliminary prediction image [i] * second weight data [i] * first weight data)

In an embodiment, the optical flow decoder 450 may output a plurality of third weight data corresponding to a result of multiplying the plurality of second weight data and the first weight data in Equation 2.

For example, the third weight data may be expressed by Equation 3 below.

[Equation 3]

3rd weight data [i] = 2nd weight data [i] * 1st weight data

The optical flow decoder 450 may process feature data of the current optical flow to obtain a plurality of third weight data and provide the plurality of third weight data to the weight application unit 632 .

The weight application unit 632 obtains a plurality of transformed prediction images by applying a plurality of third weight data to a plurality of preliminary prediction images, and the combining unit 633 combines the plurality of transformed prediction images to obtain a final prediction image. can be obtained.

When the optical flow decoder 450 outputs a plurality of third weight data, the configuration of the prediction decoding unit 630 may be the same as that of the prediction decoding unit 630 - 3 shown in FIG. 11 . In this case, the optical flow decoder 450 of FIG. 11 may output a plurality of third weight data instead of a plurality of second weight data.

13 is a flowchart of an image decoding method according to an embodiment.

Referring to FIG. 13 , in step S1310, the image decoding apparatus 600 obtains feature data of the current optical flow and feature data of the current residual image from the bitstream.

In an embodiment, the image decoding apparatus 600 may obtain feature data of the current optical flow and feature data of the current residual image by performing entropy decoding, inverse quantization, and/or inverse transformation on bits included in the bitstream. there is.

In one embodiment, the image decoding apparatus 600 may further obtain feature data of the weight data from the bitstream.

In step S1320, the image decoding apparatus 600 obtains the current optical flow and first weight data by applying feature data of the current optical flow to the neural network-based optical flow decoder 450.

In one embodiment, the image decoding apparatus 600 may obtain first weight data by applying feature data of the weight data to the weight decoder 634 .

In step S1330, the image decoding apparatus 600 acquires a current residual image by applying feature data of the current residual image to the neural network-based residual decoder 470.

In step S1340, the image decoding apparatus 600 obtains a preliminary prediction image from a previous reconstructed image based on the current optical flow. Warping may be used to obtain a preliminary prediction image.

In step S1350, the image decoding apparatus 600 obtains a final predicted image by applying first weight data to the preliminary predicted image.

In an embodiment, the image decoding apparatus 600 may obtain a final prediction image by multiplying each sample value of the preliminary prediction image by each sample value of the first weight data.

In step S1360, the image decoding apparatus 600 may obtain a current reconstructed image corresponding to the current image by combining the final prediction image and the current residual image.

In an embodiment, the image decoding apparatus 600 may obtain a current reconstructed image by adding each sample value of the current residual image to each of the sample values of the final prediction image.

14 is a flowchart of an image decoding method according to an embodiment.

Referring to FIG. 14 , in step S1410, the image decoding apparatus 600 obtains feature data of the current optical flow and feature data of the current residual image from the bitstream.

In an embodiment, the image decoding apparatus 600 may obtain feature data of the current optical flow and feature data of the current residual image by performing entropy decoding, inverse quantization, and/or inverse transformation on bits included in the bitstream. there is.

In one embodiment, the image decoding apparatus 600 may further obtain feature data of the weight data from the bitstream.

In step S1420, the image decoding apparatus 600 obtains a plurality of current optical flows and a plurality of second weight data by applying feature data of the current optical flow to the neural network-based optical flow decoder 450.

In one embodiment, the image decoding apparatus 600 may obtain a plurality of second weight data by applying feature data of the weight data to at least one weight decoder 634 .

In step S1430, the image decoding apparatus 600 obtains a current residual image by applying feature data of the current residual image to the neural network-based residual decoder 470.

In step S1440, the image decoding apparatus 600 obtains a plurality of preliminary prediction images from previous reconstructed images based on a plurality of current optical flows. Warping may be used to obtain a plurality of preliminary prediction images.

In step S1450, the image decoding apparatus 600 obtains a plurality of modified prediction images by applying a plurality of second weight data to a plurality of preliminary prediction images.

In an embodiment, the image decoding apparatus 600 may obtain a plurality of modified prediction images by multiplying each of sample values of a plurality of preliminary prediction images by each sample value of a plurality of second weight data.

In step S1460, the image decoding apparatus 600 obtains a final predicted image by combining a plurality of transformed predicted images.

In an embodiment, the image decoding apparatus 600 may obtain a final predicted image by adding sample values at the same location included in a plurality of transformed predicted images.

In step S1470, the image decoding apparatus 600 may obtain a current reconstructed image corresponding to the current image by combining the final prediction image and the current residual image.

In an embodiment, the image decoding apparatus 600 may obtain a current reconstructed image by adding each sample value of the current residual image to each of the sample values of the final prediction image.

15 is a flowchart of an image decoding method according to an embodiment.

Referring to FIG. 15 , in step S1510, the image decoding apparatus 600 obtains feature data of the current optical flow and feature data of the current residual image from the bitstream.

In an embodiment, the image decoding apparatus 600 may obtain feature data of the current optical flow and feature data of the current residual image by performing entropy decoding, inverse quantization, and/or inverse transformation on bits included in the bitstream. there is.

In one embodiment, the image decoding apparatus 600 may further obtain feature data of the weight data from the bitstream.

In step S1520, the image decoding apparatus 600 obtains a plurality of current optical flows, first weight data, and a plurality of second weight data by applying feature data of the current optical flow to the neural network-based optical flow decoder 450. .

In one embodiment, the image decoding apparatus 600 may obtain first weight data and a plurality of second weight data by applying feature data of weight data to at least one weight decoder 634 .

In step S1530, the image decoding apparatus 600 obtains a current residual image by applying feature data of the current residual image to the neural network-based residual decoder 470.

In step S1540, the image decoding apparatus 600 obtains a plurality of preliminary prediction images from previous reconstructed images based on a plurality of current optical flows. Warping may be used to obtain a plurality of preliminary prediction images.

In step S1550, the image decoding apparatus 600 obtains a plurality of modified prediction images by applying a plurality of second weight data to a plurality of preliminary prediction images.

In an embodiment, the image decoding apparatus 600 may obtain a plurality of modified prediction images by multiplying each of sample values of a plurality of preliminary prediction images by each sample value of a plurality of second weight data.

In step S1560, the image decoding apparatus 600 obtains an intermediate prediction image by combining a plurality of modified prediction images.

In an embodiment, the image decoding apparatus 600 may obtain an intermediate predicted image by adding sample values at the same position of a plurality of modified predicted images.

In step S1570, the image decoding apparatus 600 obtains a final predicted image by applying first weight data to the intermediate predicted image.

In an embodiment, the image decoding apparatus 600 may obtain a final prediction image by multiplying each sample value of the intermediate prediction image by each sample value of the first weight data.

In step S1580, the image decoding apparatus 600 may obtain a current reconstructed image corresponding to the current image by combining the final prediction image and the current residual image.

In an embodiment, the image decoding apparatus 600 may obtain a current reconstructed image by adding each sample value of the current residual image to each of the sample values of the final prediction image.

Hereinafter, the operation of the video encoding apparatus will be described with reference to FIGS. 16 to 22 .

16 is a diagram illustrating a configuration of an image encoding apparatus 1600 according to an embodiment.

Referring to FIG. 16 , an image encoding apparatus 1600 may include a predictive encoder 1610, a generator 1620, an acquirer 1630, and a predictive decoder 1640.

The predictive encoder 1610, the generator 1620, the acquirer 1630, and the predictive decoder 1640 may be implemented as a processor. The predictive encoder 1610, the generator 1620, the acquirer 1630, and the predictive decoder 1640 may operate according to instructions stored in a memory (not shown).

16 shows the prediction encoder 1610, the generator 1620, the acquisition unit 1630, and the prediction decoding unit 1640 separately, but the prediction encoder 1610, the generator 1620, and the acquisition unit. 1630 and the prediction decoding unit 1640 may be implemented through one processor. In this case, the prediction encoding unit 1610, the generation unit 1620, the acquisition unit 1630, and the prediction decoding unit 1640 may be implemented as a dedicated processor, an application processor (AP), a central processing unit (CPU), or a GPU ( It may be implemented through a combination of a general-purpose processor such as a graphic processing unit) and software. In addition, a dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processing unit for using an external memory.

The predictive encoder 1610, the generator 1620, the acquirer 1630, and the predictive decoder 1640 may include a plurality of processors. In this case, the prediction encoding unit 1610, the generation unit 1620, the acquisition unit 1630, and the prediction decoding unit 1640 are implemented as a combination of dedicated processors, or a plurality of general-purpose processors such as APs, CPUs or GPUs. It may be implemented through a combination of software.

The predictive encoder 1610 may obtain feature data of the current optical flow and feature data of the current residual image by using the current image and the previous reconstructed image.

The prediction encoder 1610 may use the neural network-based optical flow encoder 410 and the residual encoder 430 to acquire feature data of the current optical flow and feature data of the current residual image. The operation of the predictive encoder 1610 will be described in detail with reference to FIGS. 17 and 18 .

Feature data of the current optical flow and feature data of the current residual image acquired by the predictive encoder 1610 may be transmitted to the generator 1620 .

The generation unit 1620 may generate a bitstream including feature data of the current optical flow and feature data of the current residual image.

In an embodiment, the generation unit 1620 may generate a first bitstream corresponding to feature data of the current optical flow and a second bitstream corresponding to feature data of the current residual image.

The bitstream may be transmitted from the video decoding apparatus 600 through a network. Also, in one embodiment, the bitstream is a magnetic medium such as a hard disk, floppy disk and magnetic tape, an optical recording medium such as CD-ROM and DVD, and a magneto-optical medium such as a floptical disk. It may be recorded on a data storage medium including an optical medium).

The acquisition unit 1630 may acquire feature data of the current optical flow and feature data of the current residual image from the bitstream generated by the generator 1620 .

In an embodiment, the acquisition unit 1630 may receive feature data of the current optical flow and feature data of the current residual image from the predictive encoder 1610 .

Feature data of the current optical flow and feature data of the current residual image may be transferred to the prediction decoding unit 1640 .

The predictive decoding unit 1640 may obtain a current reconstructed image using feature data of the current optical flow and feature data of the current residual image.

The current reconstructed image obtained by the predictive decoding unit 1640 may be used in the encoding process of the next image.

Configurations and operations of the acquisition unit 1630 and the prediction decoding unit 1640 may be the same as those of the acquisition unit 610 and the prediction decoding unit 630 of the image decoding apparatus 600 .

For example, the configuration and operation of the acquisition unit 1630 is the same as that of the acquisition unit 610 shown in FIG. 7, and the configuration and operation of the prediction decoding unit 1640 perform the prediction decoding shown in FIGS. It may be the same as any one of the parts 630, 630-1, 630-2, 630-3, and 630-4.

17 is a diagram showing the configuration of a predictive encoder 1610 according to an embodiment.

Referring to FIG. 17 , a predictive encoder 1610 may include an optical flow encoder 410, a residual encoder 430, and a subtractor 1611.

The optical flow encoder 410 and the residual encoder 430 may be implemented as a neural network including at least one layer (eg, a convolutional layer).

The optical flow encoder 410 and the residual encoder 430 may be stored in memory. The optical flow encoder 410 and the residual encoder 430 may be implemented with at least one dedicated processor for AI.

The current image and the previous reconstructed image may be input to the optical flow encoder 410 . The optical flow encoder 410 may obtain feature data of the current optical flow by processing the current image and the previous reconstructed image according to parameters set through training. Characteristic data of the current optical flow may be transmitted to the generating unit 1620 .

A final predicted image generated from a previous reconstructed image based on feature data of the current optical flow may be input to the subtraction unit 1611 together with the current image. The final prediction image may be generated by the prediction decoding unit 1640 and transmitted to the prediction encoding unit 1610. Since the process of generating the final prediction image by the prediction decoding unit 1640 has been described with reference to FIGS. 8 to 12, the description thereof is omitted here.

The subtractor 1611 may obtain a current residual image corresponding to a difference between the current image and the final predicted image.

In an embodiment, the subtractor 1611 may obtain a current residual image by subtracting sample values of the final prediction image from sample values of the current image.

The current residual image may be input to the residual encoder 430 . The residual encoder 430 may obtain feature data of the current residual image by processing the current residual image according to parameters set through training. Feature data of the current residual image may be transmitted to the generation unit 1620 .

According to the predictive encoder 1610 shown in FIG. 17, since a bitstream including feature data of the current optical flow and feature data of the current residual image is generated, the image decoding apparatus 600 may It may include any one of the prediction decoding units 630, 630-1, 630-3, and 630-4 shown in FIGS. 11 and 12 .

18 is a diagram showing the configuration of a predictive encoder 1610-1 according to an embodiment.

Referring to FIG. 18 , a prediction encoder 1610-1 may include an optical flow encoder 410, a residual encoder 430, a weight encoder 1612, and a subtraction unit 1611.

The optical flow encoder 410, the residual encoder 430, and the weight encoder 1612 may be implemented as a neural network including at least one layer (eg, a convolutional layer).

Optical flow encoder 410, residual encoder 430, and weight encoder 1612 may be stored in memory. The optical flow encoder 410, the residual encoder 430, and the weight encoder 1612 may be implemented as at least one dedicated processor for AI.

The current image and the previous reconstructed image may be input to the optical flow encoder 410 . The optical flow encoder 410 may obtain feature data of the current optical flow by processing the current image and the previous reconstructed image according to parameters set through training. Characteristic data of the current optical flow may be transmitted to the generating unit 1620 .

The current image and the previous reconstructed image may be input to the weight encoder 1612 . The weight encoder 1612 may obtain feature data of the weight data by processing the current image and the previous reconstructed image according to parameters set through training. Feature data of the weight data may be transmitted to the generating unit 1620 .

A final predicted image generated from a previous reconstructed image based on the feature data of the current optical flow and the feature data of the weight data may be input to the subtraction unit 1611 together with the current image. The final prediction image may be generated by the prediction decoding unit 1640 and transmitted to the prediction encoding unit 1610.

The subtractor 1611 may obtain a current residual image corresponding to a difference between the current image and the final predicted image. In an embodiment, the subtractor 1611 may obtain a current residual image by subtracting sample values of the final prediction image from sample values of the current image.

The current residual image may be input to the residual encoder 430 . The residual encoder 430 may obtain feature data of the current residual image by processing the current residual image according to parameters set through training. Feature data of the current residual image may be transmitted to the generation unit 1620 .

According to the predictive encoder 1610-1 shown in FIG. 18, since a bitstream including feature data of the current optical flow, feature data of the current residual image, and feature data of weight data is generated, the image decoding apparatus 600 may include a prediction decoding unit 630 that requires feature data of weight data to generate a final prediction image, for example, the prediction decoding unit 630-2 shown in FIG. 10 .

As described above, the prediction decoding unit 630 of the image decoding apparatus 600 may obtain first weight data and/or a plurality of second weight data by using feature data of the weight data.

19 is a diagram illustrating a configuration of a generating unit 1620 according to an embodiment.

Referring to FIG. 19 , a generation unit 1620 may include a quantization unit 1621 and an entropy encoding unit 1623.

The quantization unit 1621 may quantize feature data of the current optical flow and feature data of the current residual image.

The entropy encoding unit 1623 may generate a bitstream by entropy-coding the quantized feature data of the current optical flow and the quantized feature data of the current residual image.

In one embodiment, the generation unit 1620 may further include a conversion unit. The transform unit may convert the feature data of the current optical flow and the feature data of the current residual image from the spatial domain to the frequency domain and provide them to the quantization unit 1621 .

In one embodiment, the generation unit 1620 may not include the quantization unit 1621. That is, a bitstream including feature data of the current optical flow and feature data of the current residual image may be obtained through processing by the entropy encoder 1623 .

Also, in an embodiment, the generation unit 1620 may generate a bitstream by performing binarization on feature data of the current optical flow and feature data of the current residual image. That is, when the generation unit 1620 only performs binarization, the quantization unit 1621 and the entropy encoding unit 1623 may not be included in the generation unit 1620.

In one embodiment, the generation unit 1620 applies at least one of transform, quantization, or entropy encoding to the feature data of the weight data transmitted from the prediction encoder 1610-1 to apply at least one bit including the feature data of the weight data. streams can be created.

20 is a flowchart of an image encoding method according to an embodiment.

Referring to FIG. 20 , in step S2010, the image encoding apparatus 1600 obtains feature data of the current optical flow by applying the current image and the previous reconstructed image to the optical flow encoder 410.

In an embodiment, the image encoding apparatus 1600 may obtain feature data of the weight data by applying the current image and the previous reconstructed image to the weight encoder 1612 .

In step S2020, the image encoding apparatus 1600 obtains the current optical flow and first weight data by applying feature data of the current optical flow to the optical flow decoder 450.

In one embodiment, the image encoding apparatus 1600 obtains the current optical flow by applying the feature data of the current optical flow to the optical flow decoder 450, and applies the feature data of the weight data to the weight decoder 634 to obtain a second 1 Weight data can be obtained.

In step S2030, the image encoding apparatus 1600 obtains a preliminary prediction image from a previous reconstructed image based on the current optical flow. Warping may be used to obtain a preliminary prediction image.

In step S2040, the image encoding apparatus 1600 obtains a final predicted image by applying first weight data to the preliminary predicted image.

In an embodiment, the image encoding apparatus 1600 may obtain a final prediction image by multiplying each sample value of the preliminary prediction image by each sample value of the first weight data.

In step S2050, the image encoding apparatus 1600 obtains a current residual image using the final predicted image and the current image.

In an embodiment, the image encoding apparatus 1600 may obtain a current residual image by subtracting each sample value of the final predicted image from each sample value of the current image.

In step S2060, the image encoding apparatus 1600 obtains feature data of the current residual image by applying the current residual image to the residual encoder 430.

In step S2070, the image encoding apparatus 1600 generates a bitstream including feature data of the current optical flow and feature data of the current residual image. In one embodiment, the bitstream may further include feature data of weight data.

In an embodiment, the image encoding apparatus 1600 may perform at least one of transformation, quantization, or entropy encoding on feature data of the current optical flow and feature data of the current residual image to generate a bitstream.

The process described with reference to FIG. 20 involves using any one of the prediction encoders 1610 and 1610-1 shown in FIGS. 17 and 18 and the prediction decoders 630 and 630-1 shown in FIGS. 8, 9 and 10. 1, 630-2) may be performed by the video encoding apparatus 1600 including any one of the prediction decoding unit 1640.

21 is a flowchart of an image encoding method according to an embodiment.

Referring to FIG. 21 , in step S2110, the image encoding apparatus 1600 obtains feature data of the current optical flow by applying the current image and the previous reconstructed image to the optical flow encoder 410.

In an embodiment, the image encoding apparatus 1600 may obtain feature data of the weight data by applying the current image and the previous reconstructed image to the weight encoder 1612 .

In step S2120, the image encoding apparatus 1600 obtains a plurality of current optical flows and a plurality of second weight data by applying feature data of the current optical flow to the optical flow decoder 450.

In one embodiment, the image encoding apparatus 1600 obtains a plurality of current optical flows by applying feature data of the current optical flow to the optical flow decoder 450, and applies feature data of the weight data to the weight decoder 634 Thus, a plurality of second weight data may be obtained.

In step S2130, the image encoding apparatus 1600 obtains a plurality of preliminary prediction images from previous reconstructed images based on a plurality of current optical flows. Warping may be used to obtain a plurality of preliminary prediction images.

In step S2140, the image encoding apparatus 1600 obtains a plurality of deformed prediction images by applying a plurality of second weight data to a plurality of preliminary prediction images.

In an embodiment, the image encoding apparatus 1600 may obtain a plurality of modified prediction images by multiplying each of sample values of a plurality of preliminary prediction images by each sample value of a plurality of second weight data.

In step S2150, the image encoding apparatus 1600 obtains a final predicted image by combining a plurality of modified predicted images.

In an embodiment, the image encoding apparatus 1600 may obtain a final prediction image by summing sample values at the same position of a plurality of modified prediction images.

In step S2160, the image encoding apparatus 1600 obtains a current residual image using the final predicted image and the current image.

In an embodiment, the image encoding apparatus 1600 may obtain a current residual image by subtracting each sample value of the final predicted image from each sample value of the current image.

In step S2170, the image encoding apparatus 1600 obtains feature data of the current residual image by applying the current residual image to the residual encoder 430.

In step S2180, the image encoding apparatus 1600 generates a bitstream including feature data of the current optical flow and feature data of the current residual image.

In one embodiment, the bitstream may further include feature data of weight data.

In an embodiment, the image encoding apparatus 1600 may perform at least one of transformation, quantization, or entropy encoding on feature data of the current optical flow and feature data of the current residual image to generate a bitstream.

In the process described with reference to FIG. 21, one of the prediction encoders 1610 and 1610-1 shown in FIGS. 17 and 18 and the prediction decoding unit 630-3 shown in FIG. 11 are used as a prediction decoding unit 1640. ).

22 is a flowchart of an image encoding method according to an embodiment.

Referring to FIG. 22 , in step S2210, the image encoding apparatus 1600 obtains feature data of the current optical flow by applying the current image and the previous reconstructed image to the optical flow encoder 410.

In an embodiment, the image encoding apparatus 1600 may obtain feature data of the weight data by applying the current image and the previous reconstructed image to the weight encoder 1612 .

In step S2220, the image encoding apparatus 1600 applies feature data of the current optical flow to the optical flow decoder 450 to obtain a plurality of current optical flows, first weight data, and a plurality of second weight data.

In one embodiment, the image encoding apparatus 1600 obtains a plurality of current optical flows by applying feature data of the current optical flow to the optical flow decoder 450, and applies feature data of the weight data to the weight decoder 634 Thus, first weight data and a plurality of second weight data may be obtained.

In step S2230, the image encoding apparatus 1600 obtains a plurality of preliminary prediction images from previous reconstructed images based on a plurality of current optical flows. Warping may be used to obtain a plurality of preliminary prediction images.

In step S2240, the image encoding apparatus 1600 obtains a plurality of deformed prediction images by applying a plurality of second weight data to a plurality of preliminary prediction images.

In an embodiment, the image encoding apparatus 1600 may obtain a plurality of modified prediction images by multiplying each of sample values of a plurality of preliminary prediction images by each sample value of a plurality of second weight data.

In step S2250, the image encoding apparatus 1600 obtains an intermediate prediction image by combining a plurality of modified prediction images.

In an embodiment, the image encoding apparatus 1600 may obtain an intermediate prediction image by summing sample values at the same position of a plurality of modified prediction images.

In step S2260, the image encoding apparatus 1600 obtains a final predicted image by applying first weight data to the intermediate predicted image.

In an embodiment, the image encoding apparatus 1600 may obtain a final prediction image by multiplying each sample value of the intermediate prediction image by each sample value of the first weight data.

In step S2270, the image encoding apparatus 1600 obtains a current residual image using the final predicted image and the current image.

In an embodiment, the image encoding apparatus 1600 may obtain a current residual image by subtracting each sample value of the final predicted image from each sample value of the current image.

In step S2280, the image encoding apparatus 1600 obtains feature data of the current residual image by applying the current residual image to the residual encoder 430.

In step S2290, the image encoding apparatus 1600 generates a bitstream including feature data of the current optical flow and feature data of the current residual image.

In one embodiment, the bitstream may further include feature data of weight data.

In an embodiment, the image encoding apparatus 1600 may perform at least one of transformation, quantization, or entropy encoding on feature data of the current optical flow and feature data of the current residual image to generate a bitstream.

In the process described with reference to FIG. 22, one of the prediction encoders 1610 and 1610-1 shown in FIGS. 17 and 18 and the prediction decoding unit 630-4 shown in FIG. 12 are used as a prediction decoding unit 1640. ).

At least one of the aforementioned optical flow encoder 410, residual encoder 430, optical flow decoder 450, residual decoder 470, weight encoder 1612, or weight decoder 634 may include a convolution layer. can

23 for a structure that at least one of the optical flow encoder 410, the residual encoder 430, the optical flow decoder 450, the residual decoder 470, the weight encoder 1612, or the weight decoder 634 may have. refer to and explain.

23 is a diagram illustrating the structure of a neural network 2300 according to an embodiment.

As shown in FIG. 23 , input data 2305 is input to the first convolution layer 2310 . Here, input data 2305 is input by neural network 2300 to optical flow encoder 410, residual encoder 430, optical flow decoder 450, residual decoder 470, weight encoder 1612, and weight decoder 634. Depends on which of them.

In one embodiment, when the neural network 2300 is an optical flow encoder 410, the input data 2305 is a current image and a previous reconstructed image, and when the neural network 2300 is a residual decoder 470, the input data 2305 ) may be feature data of the current residual image.

3X3X4 displayed in the first convolution layer 2310 shown in FIG. 23 illustrates convolution processing on one input data 2305 using four filter kernels of 3x3 size. As a result of the convolution process, 4 feature maps are generated by 4 filter kernels.

Feature maps generated by the first convolutional layer 2310 represent unique characteristics of the input data 2305 . For example, each feature map may indicate a vertical direction characteristic, a horizontal direction characteristic, or an edge characteristic of the input data 2305 .

Referring to FIG. 24 , the convolution operation in the first convolution layer 2310 will be described in detail.

One feature map ( 2450) can be created. Since four filter kernels are used in the first convolution layer 2310, four feature maps may be generated through a convolution operation process using the four filter kernels.

In FIG. 24, I1 to I49 indicated in the input data 2305 indicate samples of the input data 2305, and F1 to F9 indicated in the filter kernel 2430 are samples of the filter kernel 2430 (can also be referred to as parameters). represent them Also, M1 to M9 displayed on the feature map 2450 represent samples of the feature map.

In the convolution operation process, sample values of I1, I2, I3, I8, I9, I10, I15, I16, and I17 of the input data 2305 and F1, F2, F3, F4, and F5 of the filter kernel 2430, A multiplication operation of each of F6, F7, F8, and F9 is performed, and a value obtained by combining (eg, an addition operation) result values of the multiplication operation may be assigned as a value of M1 of the feature map 2450. If the stride of the convolution operation is 2, each of the sample values of I3, I4, I5, I10, I11, I12, I17, I18, I19 of the input data 2305 and F1, F2 of the filter kernel 2430 Each multiplication operation of F3, F4, F5, F6, F7, F8, and F9 is performed, and a value obtained by combining the resultant values of the multiplication operation may be assigned as the value of M2 of the feature map 2450.

While the filter kernel 2430 moves according to the stride until it reaches the last sample of the input data 2305, a convolution operation is performed between the sample values in the input data 2305 and the samples of the filter kernel 2430. , a feature map 2450 having a predetermined size may be obtained.

According to the present disclosure, parameters of the neural network 2300 through training of the neural network 2300, for example, samples of the filter kernel 2430 used in convolutional layers of the neural network 2300 (eg, , the values of F1, F2, F3, F4, F5, F6, F7, F8 and F9) of the filter kernel can be optimized.

The convolution layers included in the neural network 2300 may perform processing according to the convolution operation process described in relation to FIG. 24 , but the convolution operation process described in FIG. 24 is only an example and is not limited thereto.

Referring back to FIG. 23 , feature maps of the first convolution layer 2310 are input to the first activation layer 2320 .

The first activation layer 2320 may assign non-linear characteristics to each feature map. The first activation layer 2320 may include, but is not limited to, a sigmoid function, a Tanh function, a Rectified Linear Unit (ReLU) function, and the like.

Giving nonlinear characteristics in the first activation layer 2320 means changing and outputting some sample values of feature maps. At this time, the change is performed by applying nonlinear characteristics.

The first activation layer 2320 determines whether to transfer the sample values of the feature map to the second convolution layer 2330. For example, certain sample values among sample values of the feature map are activated by the first activation layer 2320 and transferred to the second convolution layer 2330, and certain sample values are activated by the first activation layer 2320. It is inactivated and not passed to the second convolution layer 2330. Unique characteristics of the input data 2305 indicated by the feature maps may be emphasized by the first activation layer 2320 .

The feature maps 2325 output from the first activation layer 2320 are input to the second convolution layer 2330. One of the feature maps 2325 shown in FIG. 23 may be a result of processing the feature map 2450 described with reference to FIG. 24 in the first activation layer 2320 .

3X3X4 displayed on the second convolution layer 2330 exemplifies convolution processing of the input feature maps using four 3x3 filter kernels. The output of the second convolution layer 2330 is input to the second activation layer 2340. The second activation layer 2340 may assign nonlinear characteristics to input feature maps.

The feature maps 2345 output from the second activation layer 2340 are input to the third convolution layer 2350. 3X3X1 displayed on the third convolution layer 2350 illustrates convolution processing to generate one output data 2355 using one filter kernel having a size of 3x3.

The output data 2355 is output by the neural network 2300 to any one of the optical flow encoder 410, the residual encoder 430, the optical flow decoder 450, the residual decoder 470, the weight encoder 1612, and the weight decoder 634. Depends on whether

For example, when the neural network 2300 is the residual encoder 430, the output data 2355 is feature data of the current residual image, and when the neural network 2300 is the optical flow encoder 410, the output data 2355 is It may be characteristic data of the current optical flow.

In one embodiment, the number of output data 2355 may be adjusted by adjusting the number of filter kernels used in the third convolution layer 2350 . For example, if the neural network 2300 is the optical flow decoder 450 and the number of filter kernels used in the third convolution layer 2350 is 2, the output data 2355 is the current optical flow and the first weight may contain data. Also, for example, if the neural network 2300 is the optical flow decoder 450 and the number of filter kernels used in the third convolution layer 2350 is 10, the output data 2355 is 5 current optical flows and five pieces of second weight data.

That is, even if the neural network 2300 corresponds to any of the optical flow encoder 410, the residual encoder 430, the optical flow decoder 450, the residual decoder 470, the weight encoder 1612, and the weight decoder 634, the first The number of output data 2355 may be adjusted by adjusting the number of filter kernels used in the 3 convolution layer 2350 .

23 shows that the neural network 2300 includes three convolution layers 2310, 2330, and 2350 and two activation layers 2320 and 2340, but this is only an example, and in one embodiment , the number of convolution layers and activation layers included in the neural network 2300 may be variously changed.

Also, in one embodiment, the neural network 2300 may be implemented through a recurrent neural network (RNN). In this case, it means that the CNN structure of the neural network 2300 is changed to an RNN structure.

In an embodiment, the image decoding apparatus 600 and the image encoding apparatus 1600 may include at least one arithmetic logic unit (ALU) for the above-described convolution operation and activation layer operation.

An ALU may be implemented as a processor. For the convolution operation, the ALU may include a multiplier that performs a multiplication operation between the sample values of the feature map output from the input data 2305 or the previous layer and the sample values of the filter kernel, and an adder that adds the resultant values of the multiplication. there is.

For the operation of the activation layer, the ALU is a multiplier that multiplies the input sample value with a weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and compares the multiplication result with a predetermined value to transfer the input sample value to the next layer. It may include a comparator for determining whether to transfer to .

Hereinafter, with reference to FIGS. 25 and 26, a method for training the neural networks 2300 used in the process of encoding and decoding an image will be described.

25 is a diagram for explaining a training method of the optical flow encoder 410, the residual encoder 430, the optical flow decoder 450, and the residual decoder 470.

Although the weight encoder 1612 and the weight decoder 634 are not shown in FIG. 25, the weight encoder 1612 and the weight decoder 634 also include the optical flow encoder 410, the residual encoder 430, and the optical flow decoder 450. ) and residual decoder 470.

25 shows that weight data t is output from the optical flow decoder 450, the feature data w of the current optical flow is processed by the weight decoder 634, and the weight decoder When the weight data t is output from 634, the weight decoder 634 may also be trained according to a training method to be described with reference to FIG. 25.

In one embodiment, FIG. 25 shows that feature data w of the current optical flow is output as the current training image 2520 and the previous reconstructed training image 2540 are processed by the optical flow encoder 410, However, the current training image 2520 and the previous reconstructed training image 2540 are processed by the weight encoder 1612 to output feature data of the weight data. In addition, feature data of the weight data may be processed by the weight decoder 634 to output weight data t. In this case, the weight encoder 1612 and the weight decoder 634 may also be trained according to a training method to be described with reference to FIG. 25 .

In FIG. 25 , a current training image 2520, a previous reconstructed training image 2540, and a current reconstructed training image 2560 may correspond to the aforementioned current image, previous reconstructed image, and current reconstructed image, respectively.

How similar the current reconstructed training image 2560 is to the current training image 2520 in training the optical flow encoder 410, the residual encoder 430, the optical flow decoder 450, and the residual decoder 470 , and how large the bitrate of the bitstream generated through encoding of the current training image 2520 or the sample values of the residual training image r may be considered.

To this end, in an embodiment, at least one of the first loss information 2510, the second loss information 2530, and the third loss information 2550 is calculated, and according to the calculated loss information, the optical flow encoder 410, Residual encoder 430, optical flow decoder 450, and residual decoder 470 may be trained.

Referring to FIG. 25 , a current training image 2520 and a previous reconstructed training image 2540 may be input to the optical flow encoder 410 . The optical flow encoder 410 may output feature data w of the current optical flow by processing the current training image 2520 and the previous reconstructed training image 2540 .

In one embodiment, the current training image 2520 and the previous reconstructed training image 2540 may be input to the weight encoder 1612, and feature data of the weight data may be output by the weight encoder 1612.

Feature data (w) of the current optical flow may be input to the optical flow decoder 450, and the optical flow decoder 450 processes the feature data (w) of the current optical flow to obtain the current optical flow (g) and weight data (t) can be output. The weight data t may include the aforementioned first weight data or a plurality of second weight data. In one embodiment, the weight data t may include first weight data and a plurality of second weight data.

In one embodiment, the weight decoder 634 may output weight data t by processing feature data w of the current optical flow.

The final prediction image generation process 2500 is applied to the previous reconstructed training image 2540 based on the current optical flow (g) and weight data (t), so that a final prediction training image similar to the current training image 2520 ( x'') can be obtained.

The final prediction training image (x″) is a final prediction image of any one of the prediction decoding units 630, 630-1, 630-2, 630-3, and 630-4 described above with reference to FIGS. 8 to 12. It can be created according to the creation process.

For example, when the weight data t includes first weight data, a preliminary prediction training image is obtained through warping on a previous reconstruction training image 2540, and the first weight data is obtained for the preliminary prediction training image. By being applied, a final predictive training image (x'') may be obtained.

As another example, a plurality of preliminary prediction training images are acquired by performing warping on previous reconstructed training images 2540 based on a plurality of current optical flows (g), and a plurality of second prediction training images are obtained for the plurality of preliminary prediction training images. Weight data may be applied to obtain a plurality of deformed prediction training images. In addition, a plurality of deformed prediction training images may be combined to obtain a final prediction training image (x'').

A residual training image (r) corresponding to a difference between the final predicted training image (x″) and the current training image 2520 may be input to the residual encoder 430 .

The residual encoder 430 may process the residual training image r and output feature data v of the residual training image.

The residual decoder 470 may obtain a reconstructed residual training image r′ by processing feature data v of the residual training image.

A current reconstructed training image 2560 may be obtained according to a combination of the reconstructed residual training image (r') and the final predictive training image (x'').

For training of optical flow encoder 410, residual encoder 430, optical flow decoder 450 and residual decoder 470 (and weight encoder 1612 and weight decoder 634), first loss information ( 2510), at least one of the second loss information 2530 and the third loss information 2550 may be obtained.

The first loss information 2510 is sample values of the feature data w of the current optical flow, entropy of the feature data w of the current optical flow, or bitrate of a bitstream corresponding to the feature data w of the current optical flow. It can be calculated from at least one of

The second loss information 2530 may be calculated from the sample values of the residual training image r, the entropy of the feature data v of the residual training image, or the bitrate of the bitstream corresponding to the feature data v of the residual training image. can

Since the first loss information 2510 and the second loss information 2530 are related to the coding efficiency for the current training image 2520, the first loss information 2510 and the second loss information 2530 are compression loss. information may be referenced.

In one embodiment, although FIG. 25 shows that the first loss information 2510 and the second loss information 2530 related to the encoding efficiency of the current training image 2520 are derived, the current training image 2520 One piece of loss information corresponding to the bit rate of one bitstream generated through encoding may be derived.

The third loss information 2550 may correspond to a difference between the current training image 2520 and the current reconstructed training image 2560 . The difference between the current training image 2520 and the current reconstruction training image 2560 is the L1-norm value between the current training image 2520 and the current reconstruction training image 2560, the L2-norm value, SSIM (Structural Similarity) value, Peak Signal-To-Noise Ratio-Human Vision System (PSNR-HVS) value, Multiscale SSIM (MS-SSIM) value, Variance Inflation Factor (VIF) value, or Video Multimethod Assessment Fusion (VMAF) value. can include

Since the third loss information 2550 is related to the quality of the current reconstructed training image 2560, the third loss information 2550 may be referred to as quality loss information.

Optical flow encoder 410, residual encoder 430, optical flow decoder 450 and residual decoder 470 (and weight encoder 1612 and weight decoder 634) provide first loss information 2510, The final loss information derived from at least one of the second loss information 2530 and the third loss information 2550 may be trained to be reduced or minimized.

In one embodiment, the optical flow encoder 410, the residual encoder 430, the optical flow decoder 450 and the residual decoder 470 (and the weight encoder 1612 and the weight decoder 634) are used to determine the parameters of the preset parameters. By changing the value, the final loss information can be reduced or minimized.

In one embodiment, the final loss information may be calculated according to Equation 4 below.

[Equation 4]

Final loss information = a*first loss information+b*second loss information+c*third loss information

In Equation 4, a, b, and c are weights applied to the first loss information 2510, the second loss information 2530, and the third loss information 2550, respectively.

According to Equation 4, the optical flow encoder 410, the residual encoder 430, the optical flow decoder 450, and the residual decoder 470 (and the weight encoder 1612 and the weight decoder 634) perform the current reconstruction It can be seen that training is performed in a direction in which the training image 2560 is most similar to the current training image 2520 and the size of the bitstream corresponding to the data output from the optical flow encoder 410 and the residual encoder 430 is minimized. .

26 is a diagram for explaining a training process of the optical flow encoder 410, the residual encoder 430, the optical flow decoder 450, and the residual decoder 470 by the training apparatus 2600.

Although not shown in FIG. 26 , the weight encoder 1612 and/or the weight decoder 634 may also be trained according to the process shown in FIG. 26 .

The training process described with reference to FIG. 25 may be performed by the training device 2600 . The training device 2600 may be, for example, the video encoding device 1600 or a separate server. Parameters obtained as a result of training may be stored in the image encoding device 1600 and the image decoding device 600 .

Referring to FIG. 26, a training device 2600 includes an optical flow encoder 410, a residual encoder 430, an optical flow decoder 450, and a residual decoder 470 (and a weight encoder 1612 and/or a weight decoder). Parameters of (634)) may be initially set (S2610). Thereby, the optical flow encoder 410, the residual encoder 430, the optical flow decoder 450, and the residual decoder 470 (and the weight encoder 1612 and/or the weight decoder 634) are initially set parameters can operate according to

The training device 2600 may input the current training image 2520 and the previous reconstructed training image 2540 to the optical flow encoder 410 (S2615).

The optical flow encoder 410 processes the current training image 2520 and the previous reconstructed training image 2540 and outputs feature data w of the current optical flow to the training device 2600 and the optical flow decoder 450. It can (S2620).

In one embodiment, feature data w of the current optical flow may be input to the weight decoder 634 .

In addition, in one embodiment, the current training image 2520 and the previous reconstructed training image 2540 are input to the weight encoder 1612, and feature data of the weight data from the weight encoder 1612 is output to the optical flow decoder 450 Alternatively, it may be input to the weight decoder 634.

The training apparatus 2600 may calculate first loss information 2510 from feature data w of the current optical flow (S2625).

The optical flow decoder 450 may process feature data w of the current optical flow and output the current optical flow g and weight data t to the training device 2600 (S2630).

In an embodiment, the weight decoder 634 may output the weight data to the training device 2600 by processing feature data w of the current optical flow or feature data t of the weight data.

The training apparatus 2600 may generate a final prediction training image (x″) from the previous reconstructed training image 2540 using the current optical flow g and weight data t (S2635), and the final prediction A residual training image (r) may be acquired using the training image (x″) and the current training image (2520) (S2640).

The training device 2600 may input the residual training image r to the residual encoder 430 (S2645).

The residual encoder 430 may process the residual training image r and output feature data v of the residual training image to the residual decoder 470 and the training device 2600 (S2650).

The training apparatus 2600 may calculate second loss information 2530 from the residual training image r or feature data v of the residual training image (S2655).

The residual decoder 470 may process the feature data v of the residual training image and output the reconstructed residual training image r' to the training device 2600 (S2660).

The training device 2600 may obtain a current reconstructed training image 2560 by combining the reconstructed residual training image r' and the final predictive training image 2560 (S2665), and the current training image 2520 ) and the current reconstructed training image 2560, third loss information 2550 corresponding to the difference may be calculated (S2670).

The training apparatus 2600 calculates final loss information by combining at least one of the first loss information 2510, the second loss information 2530, and the third loss information 2550, and the optical flow encoder 410, the residual The encoder 430, the optical flow decoder 450, and the residual decoder 470 (and the weight encoder 1612 and/or the weight decoder 634) perform initial setting through a back propagation process based on the final loss information The set parameters may be updated (S2675, S2680, S2685, S2690).

Then, the optical flow encoder 410, the residual encoder 430, the optical flow decoder 450, and the residual decoder 470 (and the weight encoder 1612 and/or the weight decoder 634) minimize the final loss information Parameters may be updated while repeating processes S2615 to S2690 until At this time, during each iteration, the optical flow encoder 410, the residual encoder 430, the optical flow decoder 450, and the residual decoder 470 (and the weight encoder 1612 and/or the weight decoder 634) It can operate according to the parameters updated in the previous process.

In the image encoding apparatus 1600, the image decoding apparatus 600, and methods for encoding and decoding images according to an embodiment, objects included in images are occluded, have large motion, or have small motion. Even if there is, the task is to efficiently encode and decode corresponding images.

In addition, an image encoding apparatus 1600, an image decoding apparatus 600, and a method for encoding and decoding an image according to an embodiment have the task of reducing a bit rate of a bitstream generated as a result of encoding an image. do.

In addition, the image encoding apparatus 1600, the image decoding apparatus 600, and the method for encoding and decoding an image according to an embodiment prevent an increase in the bitrate of a bitstream due to a change in brightness of images. do it with

In addition, the video encoding apparatus 1600, the video decoding apparatus 600, and the video encoding and decoding method according to an embodiment have the task of providing an AI-based end-to-end encoding/decoding system. do.

An image decoding method using AI according to an embodiment may include acquiring feature data of a current optical flow and feature data of a current residual image from a bitstream (S1310).

The image decoding method may include acquiring the current optical flow and first weight data by applying feature data of the current optical flow to the optical flow decoder 450 (S1320).

The image decoding method may include obtaining a current residual image by applying feature data of the current residual image to the residual decoder 470 (S1330).

The image decoding method may include obtaining a preliminary prediction image from a previous reconstructed image based on the current optical flow (S1340).

The image decoding method may include obtaining a final prediction image by applying sample values of the first weight data to sample values of the preliminary prediction image (S1350).

The image decoding method may include acquiring a current reconstructed image corresponding to the current image by combining the final prediction image and the current residual image (S1360).

In one embodiment, the optical flow decoder 450 is trained to reduce the bitrate of a bitstream including sample values of the residual training image or feature data of the residual training image, and the residual training image is included in the final prediction image. It may correspond to a difference between the corresponding predictive training image and the current training image corresponding to the current image.

In one embodiment, as a result of training the optical flow decoder 450, the sample value in the first weight data is closer to 1 as the difference between the sample value in the preliminary prediction image and the sample value at the same position in the current image is smaller. can have a value.

In one embodiment, as a result of training the optical flow decoder 450, if a sample value in the preliminary prediction image is greater than a sample value at the same position in the current image, the sample value in the first weight data has a value less than 1. can

In one embodiment, as a result of training the optical flow decoder 450, if a sample value in the preliminary prediction image is smaller than a sample value at the same location in the current image, the sample value in the first weight data has a value greater than 1. can have

In an embodiment, as feature data of the current optical flow is processed by the optical flow decoder, a plurality of current optical flows, first weight data, and a plurality of second weight data may be obtained.

In an embodiment, the obtaining of preliminary prediction images may include acquiring a plurality of preliminary prediction images from previous reconstructed images based on a plurality of current optical flows (S1540).

In an embodiment, the obtaining of the final prediction image may include acquiring a plurality of modified prediction images by applying a plurality of second weight data to a plurality of preliminary prediction images (S1550).

In an embodiment, the obtaining of the final prediction image may include obtaining an intermediate prediction image by combining a plurality of modified prediction images (S1560).

In an embodiment, the obtaining of the final prediction image may include obtaining a final prediction image by applying sample values of the first weight data to sample values of the intermediate prediction image (S1570).

In one embodiment, the number of optical flow decoders 450 is plural, and each of the plurality of optical flow decoders may output a pair of current optical flow and second weight data.

In one embodiment, the sum of sample values at the same position in the plurality of second weight data may be 1.

The image decoding apparatus 600 using AI according to an embodiment may include an acquisition unit 610 that obtains feature data of a current optical flow and feature data of a current residual image from a bitstream.

The image decoding apparatus 600 applies feature data of a current optical flow to an optical flow decoder to obtain a current optical flow and first weight data, and applies feature data of a current residual image to a residual decoder to obtain a current residual image Obtain a preliminary prediction image from a previous reconstructed image based on the current optical flow, obtain a final prediction image by applying sample values of the first weight data to sample values of the preliminary prediction image, and obtain a final prediction image and a current residual It may include a predictive decoding unit (630; 630-1; 630-2; 630-3; 630-4) that combines images to obtain a current reconstructed image corresponding to the current image.

An image encoding method using AI according to an embodiment may include obtaining characteristic data of a current optical flow by applying a current image and a previous reconstructed image to the optical flow encoder 410 (S2010).

The image encoding method may include obtaining a current optical flow and first weight data by applying feature data of the current optical flow to the optical flow decoder 450 (S2020).

The image encoding method may include obtaining a preliminary prediction image from a previous reconstructed image based on the current optical flow (S2030).

The image encoding method may include obtaining a final prediction image by applying sample values of the first weight data to sample values of the preliminary prediction image (S2040).

The image encoding method may include acquiring a current residual image corresponding to a difference between a final predicted image and a current image (S2050).

The image encoding method may include acquiring feature data of the current residual image by applying the current residual image to the residual encoder 430 (S2060).

The image encoding method may include generating a bitstream including feature data of the current optical flow and feature data of the current residual image (S2070).

In an embodiment, as feature data of the current optical flow is processed by the optical flow decoder 450 , a plurality of current optical flows, first weight data, and a plurality of second weight data may be obtained.

In an embodiment, the obtaining of preliminary prediction images may include obtaining a plurality of preliminary prediction images from previous reconstructed images based on a plurality of current optical flows (S2230).

In an embodiment, the obtaining of the final prediction image may include acquiring a plurality of modified prediction images by applying a plurality of second weight data to the plurality of preliminary prediction images (S2240).

In an embodiment, obtaining a final predicted image may include obtaining an intermediate predicted image by combining a plurality of transformed predicted images (S2250).

In an embodiment, the obtaining of the final prediction image may include obtaining a final prediction image by applying sample values of the first weight data to sample values of the intermediate prediction image (S2260).

An image encoding apparatus 1600 using AI according to an embodiment applies a current image and a previous reconstructed image to an optical flow encoder to acquire feature data of a current optical flow, and acquires feature data of the current optical flow to an optical flow decoder. to obtain a current optical flow and first weight data, obtain a preliminary prediction image from a previous reconstructed image based on the current optical flow, and apply sample values of the first weight data to sample values of the preliminary prediction image to obtain a final A predictive encoder (1610; 1610) obtains a predicted image, obtains a current residual image corresponding to a difference between the final predicted image and the current image, and obtains feature data of the current residual image by applying the current residual image to a residual encoder. -1) may be included.

The image encoding apparatus may include a generator 1620 that generates a bitstream including feature data of the current optical flow and feature data of the current residual image.

An image decoding method using AI according to an embodiment may include obtaining feature data of a current optical flow and feature data of a current residual image from a bitstream (S1410).

The image decoding method may include obtaining a plurality of current optical flows and a plurality of second weight data by applying feature data of the current optical flow to the optical flow decoder 450 (S1420).

The image decoding method may include obtaining a current residual image by applying feature data of the current residual image to the residual decoder 470 (S1430).

The image decoding method may include acquiring a plurality of preliminary prediction images from a previous reconstructed image based on a plurality of current optical flows (S1440).

The image decoding method may include obtaining a plurality of modified prediction images by applying sample values of a plurality of second weight data to sample values of a plurality of preliminary prediction images (S1450).

The image decoding method may include obtaining a final predicted image by combining a plurality of transformed predicted images (S1460).

The image decoding method may include acquiring a current reconstructed image corresponding to the current image by combining the final prediction image and the current residual image (S1470).

An image encoding method using AI according to an embodiment may include obtaining characteristic data of a current optical flow by applying a current image and a previous reconstructed image to the optical flow encoder 410 (S2110).

The image encoding method may include obtaining a plurality of current optical flows and a plurality of second weight data by applying feature data of the current optical flow to the optical flow decoder 450 (S2120).

The image encoding method may include acquiring a plurality of preliminary prediction images from a previous reconstructed image based on a plurality of current optical flows (S2130).

The image encoding method may include obtaining a plurality of modified prediction images by applying sample values of a plurality of second weight data to sample values of a plurality of preliminary prediction images (S2140).

The image encoding method may include acquiring a final predicted image by combining a plurality of transformed predicted images (S2150).

The image encoding method may include acquiring a current residual image corresponding to a difference between a final predicted image and a current image (S2160).

The image encoding method may include acquiring feature data of the current residual image by applying the current residual image to the residual encoder 430 (S2170).

The image encoding method may include generating a bitstream including feature data of the current optical flow and feature data of the current residual image (S2180).

In the image encoding apparatus 1600, the image decoding apparatus 600, and methods for encoding and decoding images according to an embodiment, objects included in images are occluded, have large motion, or have small motion. Even if it has, it is possible to encode and decode images efficiently.

In addition, the image encoding apparatus 1600, the image decoding apparatus 600, and the method for encoding and decoding an image according to an embodiment can reduce the bit rate of a bitstream generated as a result of encoding the image.

In addition, the image encoding apparatus 1600, the image decoding apparatus 600, and methods for encoding and decoding images according to an embodiment can prevent the bitrate of a bitstream from increasing due to a change in brightness of images. .

In addition, the video encoding apparatus 1600, the video decoding apparatus 600, and the video encoding and decoding method according to an embodiment may provide an AI-based end-to-end encoding/decoding system.

On the other hand, the above-described embodiments of the present disclosure can be written as a program that can be executed on a computer, and the written program can be stored in a storage medium readable by a device.

The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary storage medium' only means that it is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to the case where data is stored semi-permanently in the storage medium and temporary It does not discriminate if it is saved as . For example, a 'non-temporary storage medium' may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store or between two user devices (eg smartphones). It can be distributed (eg downloaded or uploaded) directly or online. In the case of online distribution, at least a part of a computer program product (eg, a downloadable app) is stored on a device-readable storage medium such as a memory of a manufacturer's server, an application store's server, or a relay server. It can be temporarily stored or created temporarily.

In the above, the technical spirit of the present disclosure has been described in detail with preferred embodiments, but the technical spirit of the present disclosure is not limited to the above embodiments, and those skilled in the art within the scope of the technical spirit of the present disclosure Various modifications and changes are possible by the person.

Claims (13)

In the image decoding method using AI (Artificial Intelligence),
acquiring feature data of the current optical flow and feature data of the current residual image from the bitstream (S1310);
acquiring the current optical flow and first weight data by applying feature data of the current optical flow to the optical flow decoder 450 (S1320);
acquiring the current residual image by applying feature data of the current residual image to the residual decoder 470 (S1330);
obtaining a preliminary prediction image from a previous reconstructed image based on the current optical flow (S1340);
obtaining a final prediction unit by applying sample values of the first weight data to sample values of the preliminary prediction image (S1350); and
and acquiring a current reconstructed image corresponding to the current image by combining the final predicted image and the current residual image (S1360).
According to claim 1,
The optical flow decoder 450 is trained to reduce the bitrate of sample values of the residual training image or a bitstream including feature data of the residual training image,
The residual training image corresponds to a difference between a final predicted training image corresponding to the final predicted image and a current training image corresponding to the current image.
According to any one of claims 1 to 2,
As a result of training of the optical flow decoder 450, the sample value in the first weight data is a value closer to 1 as the difference between the sample value in the preliminary prediction image and the sample value at the same position in the current image is smaller Having, video decoding method.
According to any one of claims 1 to 3,
As a result of training the optical flow decoder 450, if the sample value in the preliminary prediction image is greater than the sample value at the same position in the current image, the sample value in the first weight data has a value smaller than 1, Video decoding method.
According to any one of claims 1 to 4,
As a result of training of the optical flow decoder 450, if the sample value in the preliminary prediction image is smaller than the sample value at the same position in the current image, the sample value in the first weight data has a value greater than 1 , Video decoding method.
According to any one of claims 1 to 5,
As the feature data of the current optical flow is processed by the optical flow decoder 450, a plurality of current optical flows, the first weight data, and a plurality of second weight data are obtained;
Obtaining the preliminary prediction image,
Acquiring a plurality of preliminary prediction images from the previous reconstructed image based on the plurality of current optical flows (S1540);
Obtaining the final predicted image,
obtaining a plurality of modified prediction images by applying the plurality of second weight data to the plurality of preliminary prediction images (S1550);
obtaining an intermediate prediction image by combining the plurality of modified prediction images (S1560); and
and obtaining the final prediction image by applying sample values of the first weight data to sample values of the intermediate prediction image (S1570).
According to any one of claims 1 to 6,
The number of optical flow decoders 450 is plural,
An image decoding method, wherein each of the plurality of optical flow decoders outputs a pair of current optical flow and second weight data.
According to any one of claims 1 to 7,
The video decoding method of claim 1, wherein the sum of sample values at the same position in the plurality of second weight data is 1.
A computer-readable recording medium on which a program for performing the method of any one of claims 1 to 8 is recorded.
In the video decoding apparatus 600 using AI,
an acquisition unit 610 that obtains feature data of a current optical flow and feature data of a current residual image from a bitstream; and
The current optical flow and first weight data are obtained by applying feature data of the current optical flow to an optical flow decoder, and the current residual image is obtained by applying feature data of the current residual image to a residual decoder. A preliminary prediction image is obtained from a previous reconstructed image based on the current optical flow, a final prediction image is obtained by applying sample values of the first weight data to sample values of the preliminary prediction image, and the final prediction image and the current A video decoding apparatus comprising: a predictive decoding unit (630; 630-1; 630-2; 630-3; 630-4) that combines residual images to obtain a current reconstructed image corresponding to a current image.
In the video encoding method using AI,
obtaining feature data of the current optical flow by applying the current image and the previous reconstructed image to the optical flow encoder 410 (S2010);
obtaining the current optical flow and first weight data by applying feature data of the current optical flow to the optical flow decoder 450 (S2020);
obtaining a preliminary prediction image from a previous reconstructed image based on the current optical flow (S2030);
obtaining a final prediction image by applying sample values of the first weight data to sample values of the preliminary prediction image (S2040);
obtaining a current residual image corresponding to a difference between the final predicted image and the current image (S2050);
obtaining feature data of the current residual image by applying the current residual image to the residual encoder 430 (S2060); and
and generating a bitstream including feature data of the current optical flow and feature data of the current residual image (S2070).
According to claim 11,
As the feature data of the current optical flow is processed by the optical flow decoder 450, a plurality of current optical flows, the first weight data, and a plurality of second weight data are obtained;
Obtaining the preliminary prediction image,
Acquiring a plurality of preliminary prediction images from the previous reconstructed image based on the plurality of current optical flows (S2230);
Obtaining the final predicted image,
acquiring a plurality of modified prediction images by applying the plurality of second weight data to the plurality of preliminary prediction images (S2240);
obtaining an intermediate prediction image by combining the plurality of modified prediction images (S2250); and
and obtaining the final prediction image by applying sample values of the first weight data to sample values of the intermediate prediction image (S2260).
In the video encoding device 1600 using AI,
Applying a current image and a previous reconstructed image to an optical flow encoder to obtain feature data of a current optical flow, and applying the feature data of the current optical flow to an optical flow decoder to obtain the current optical flow and first weight data, , Obtaining a preliminary prediction image from a previous reconstructed image based on the current optical flow, obtaining a final prediction image by applying sample values of the first weight data to sample values of the preliminary prediction image, and obtaining a final prediction image and a predictive encoder (1610; 1610-1) for acquiring a current residual image corresponding to a difference between the current images and applying the current residual image to a residual encoder to acquire feature data of the current residual image; and
and a generation unit 1620 configured to generate a bitstream including feature data of the current optical flow and feature data of the current residual image.

Images