KR20220120436A - Artificial intelligence based encoding apparatus and decoding apparatus of image, and method thereby - Google Patents

Abstract

According to one embodiment of the present disclosure is a method for restoring optical flow. The method comprises: a step of acquiring feature data on current residual optical flow from bitstream for a current image; a step of applying the feature data on the current residual optical flow to a neural network-based first decoder to acquire the current residual optical flow; a step of acquiring current prediction optical flow by using at least one of previous optical flow, feature data on the previous optical flow, and feature data on previous residual optical flow; and a step of restoring the current optical flow by using the current residual optical flow and the current prediction optical flow.

Description

AI-based video encoding and decoding apparatus, and method by the same

The present disclosure relates to encoding and decoding of an image. More specifically, the present disclosure relates to a technique for encoding and decoding an optical flow required for inter prediction of an image using artificial intelligence (AI), for example, a neural network, and a technique for encoding and decoding an image.

In codecs such as H.264 Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC), an image is divided into blocks, and each block is subjected to predictive encoding and coding through inter prediction or intra prediction. Predictive decoding is possible.

Intra prediction is a method of compressing an image by removing spatial redundancy within an image, and inter prediction is a method of compressing an image by removing temporal redundancy between images.

As a representative example of inter prediction, motion estimation coding exists. Motion estimation coding predicts blocks of a current image using a reference image. A reference block most similar to the current block may be searched for in a predetermined search range using a predetermined evaluation function. A current block is predicted based on a reference block, and a prediction block generated as a result of prediction is subtracted from the current block to generate and encode a residual block.

In order to derive a motion vector indicating a reference block in a reference image, motion vectors of previously coded blocks may be used as a prediction motion vector of the current block. A differential motion vector that is a difference between the motion vector of the current block and the predicted motion vector is signaled to the decoder side through a predetermined method.

An apparatus for encoding and decoding an image according to an embodiment, and a method by the same, have an object of enabling signaling of an optical flow required for inter prediction at a low bit rate.

In addition, an apparatus for encoding and decoding an image according to an exemplary embodiment, and a method using the same have an object of accurately reconstructing an optical flow.

In addition, an apparatus for encoding and decoding an image, and a method therefor according to an embodiment, have a technical task of accurately reconstructing an image from a bitstream having a low bitrate.

According to an embodiment, a method of restoring an optical flow using AI includes: acquiring feature data for a current residual optical flow from a bitstream for a current image; obtaining the current residual optical flow by applying feature data for the current residual optical flow to a first decoder based on a neural network; obtaining a current predicted optical flow by using at least one of a previous optical flow, feature data for the previous optical flow, and feature data for a previous residual optical flow; and reconstructing a current optical flow using the current residual optical flow and the current predicted optical flow.

In an embodiment, the current image may be reconstructed based on a current prediction image generated from a previous reconstructed image based on the reconstructed current optical flow and current residual image data.

In an embodiment, obtaining the current prediction optical flow may include determining the previous optical flow as the current prediction optical flow.

In an embodiment, the obtaining of the current prediction optical flow comprises: converting at least one of the previous optical flow, the characteristic data for the previous optical flow, and the characteristic data for the previous residual optical flow to a first prediction neural network ( first prediction neural network) to obtain the current prediction optical flow.

In an embodiment, the obtaining of the current prediction optical flow comprises: converting at least one of the previous optical flow, the characteristic data for the previous optical flow, and the characteristic data for the previous residual optical flow to a second prediction neural network ( second prediction neural network) to obtain a second-order optical flow between the current prediction optical flow and the previous optical flow; and changing the previous optical flow according to the second-order optical flow to generate the current predicted optical flow.

In an embodiment, the obtaining of the current prediction optical flow includes: obtaining feature data for a second-order optical flow between the current prediction optical flow and the previous optical flow from the bitstream; obtaining the second-order optical flow by applying the feature data for the second-order optical flow to a third decoder based on a neural network; and changing the previous optical flow according to the second-order optical flow to generate the current predicted optical flow.

In an embodiment, the feature data for the current residual optical flow may be obtained through entropy decoding and inverse quantization of the bitstream.

In one embodiment, the first decoder based on the neural network, a first loss corresponding to a difference between a current training image and a current reconstructed training image corresponding to the current training image Information; and second loss information corresponding to entropy of feature data for the current residual optical flow of the current training image.

In an embodiment, the feature data for the current residual optical flow may be obtained from the bitstream when the current image corresponds to a P frame following a P (predictive) frame.

In an embodiment, when the current image corresponds to a P frame after an I (intra) frame, the method of restoring the optical flow may include: acquiring feature data for the current optical flow from a bitstream; The method may further include applying the feature data for the current optical flow to a fourth decoder based on a neural network to restore the current optical flow.

According to an embodiment, an apparatus for restoring an optical flow using AI includes: an acquisition unit configured to acquire feature data for a current residual optical flow from a bitstream for a current image; and applying the feature data for the current residual optical flow to the first decoder based on a neural network to obtain the current residual optical flow, and to obtain the current residual optical flow, feature data for the previous optical flow, the previous optical flow, and the previous residual optical flow and a prediction decoder that obtains a current prediction optical flow by using at least one of the feature data and reconstructs the current optical flow using the current residual optical flow and the current prediction optical flow.

An optical flow encoding method using AI according to an embodiment includes: obtaining a current predicted optical flow from at least one of a previous optical flow, feature data for the previous optical flow, and feature data for a previous residual optical flow ; acquiring feature data for a current residual optical flow by applying a current image, a previous reconstructed image, and the current predicted optical flow to a first encoder based on a neural network; and generating a bitstream corresponding to feature data for the current residual optical flow, wherein the current residual optical flow may correspond to a difference between a current optical flow and the current prediction optical flow.

An apparatus for encoding an optical flow using AI according to an embodiment obtains a current prediction optical flow from at least one of a previous optical flow, feature data for the previous optical flow, and feature data for the previous residual optical flow, a prediction encoder for obtaining feature data for a current residual optical flow by applying a current image, a previous reconstructed image, and the current predicted optical flow to a first encoder based on a neural network; and a generator configured to generate a bitstream corresponding to feature data for the current residual optical flow, wherein the current residual optical flow may correspond to a difference between a current optical flow and the current predicted optical flow.

An apparatus for encoding and decoding an image according to an embodiment, and a method using the same, may enable signaling of an optical flow required for inter prediction at a low bit rate.

In addition, the apparatus for encoding and decoding an image according to an embodiment, and a method using the same may accurately restore an optical flow.

Also, the apparatus for encoding and decoding an image according to an embodiment, and a method therefor, can accurately reconstruct an image from a bitstream having a low bitrate.

1 is a diagram illustrating an AI-based inter prediction process for an image.
2 is a diagram illustrating successive images and an optical flow between successive images.
3 is a diagram illustrating a configuration of an image decoding apparatus according to an embodiment.
FIG. 4 is a diagram showing the configuration of the acquisition unit shown in FIG. 3 .
FIG. 5 is a diagram illustrating a configuration of a prediction decoding unit shown in FIG. 3 .
6 is a diagram illustrating a configuration of an optical flow prediction unit according to an embodiment.
7 is a diagram illustrating a configuration of an optical flow prediction unit according to another embodiment.
8 is a diagram illustrating a configuration of an optical flow prediction unit according to another embodiment.
9 is a flowchart of a method of restoring an optical flow according to an embodiment.
10 is a diagram illustrating another configuration of a predictive decoding unit.
11 is a flowchart of a method of restoring an optical flow according to another embodiment.
12 is a diagram illustrating a configuration of an image encoding apparatus according to an embodiment.
FIG. 13 is a diagram showing the configuration of the predictive encoder shown in FIG. 12 .
14 is a diagram illustrating a configuration of an optical flow prediction unit according to an embodiment.
FIG. 15 is a diagram showing the configuration of the generation unit shown in FIG. 12 .
16 is a flowchart of a method of encoding an optical flow according to an embodiment.
17 is a diagram showing another configuration of a predictive encoding unit.
18 is a diagram illustrating a structure of a neural network according to an embodiment.
FIG. 19 is a diagram for explaining a convolution operation in the convolution layer shown in FIG. 18 .
20 is a diagram for explaining a method of training neural networks used in an inter prediction process.
21 is a diagram for explaining a training process of a training apparatus for neural networks used in an inter prediction process.
22 is a diagram for explaining another training process of a training apparatus for neural networks used in an inter prediction process.

Since the present disclosure can make various changes and can have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail through the detailed description. However, this is not intended to limit the embodiments of the present disclosure, and it should be understood that the present disclosure includes all modifications, equivalents and substitutes included in the spirit and scope of various embodiments.

In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter of the present disclosure, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the specification are only identifiers for distinguishing one component from other components.

In addition, in this specification, when a component is referred to as "connected" or "connected" with another component, the component may be directly connected or directly connected to the other component, but in particular It should be understood that, unless there is a description to the contrary, it may be connected or connected through another element in the middle.

In addition, in the present specification, components expressed as '~ part (unit)', 'module', etc. are two or more components combined into one component, or two or more components for each more subdivided function. may be differentiated into In addition, each of the components to be described below may additionally perform some or all of the functions of other components in addition to the main functions they are responsible for, and some of the main functions of each component may be different It goes without saying that it may be performed exclusively by the component.

Also, in this specification, an 'image' may mean a still image (or frame), a moving picture composed of a plurality of continuous still images, or a video.

Also, in the present specification, a 'neural network' is a representative example of an artificial neural network model simulating a brain nerve, and is not limited to an artificial neural network model using a specific algorithm. A neural network may also be referred to as a deep neural network.

In addition, in the present specification, a 'parameter' is a value used in a calculation process of each layer constituting a neural network, and may be used, for example, when an input value is applied to a predetermined calculation expression. A parameter is a value set as a result of training, and may be updated through separate training data if necessary.

Also, in the present specification, 'feature data' refers to data obtained by processing input data by a neural network-based encoder. The feature data may be one-dimensional or two-dimensional data including several samples. Feature data may be referred to as a latent representation. The feature data represents a feature latent in data output by a decoder, which will be described later.

In addition, in this specification, 'current image' means an image that is a current processing target, 'current optical flow' means an optical flow acquired in relation to the current image, and 'current residual image data' is related to the current image It means the residual image data obtained by

In addition, in this specification, 'previous image' refers to an image to be processed before the current image, 'previous optical flow' refers to an optical flow acquired in relation to the previous image, and 'previous residual image data' refers to the previous image and It refers to residual image data obtained in relation to each other.

Also, in the present specification, a 'sample' refers to data assigned to a sampling position in an image, a feature map, or feature data, and to be processed. For example, the sample may include pixels in a two-dimensional image.

1 is a diagram illustrating an AI-based inter prediction process for an image.

1 illustrates a process of encoding and decoding a current image (x _i ), in inter prediction, the first encoder 110 , the second encoder 130 , the first decoder 150 , and the second decoder 170 are used The first encoder 110 , the second encoder 130 , the first decoder 150 , and the second decoder 170 are implemented as a neural network.

Inter prediction is a process of encoding and decoding the current image (x _i ) using temporal redundancy between the current image (x _i ) and the previous reconstructed image (y _i-1 ).

A position difference (or motion vector) between blocks or samples in the current image (x _i ) and reference blocks or reference samples in the previous reconstructed image (y _i-1 ) is used for encoding and decoding the current image (x _i ) is used for This position difference may be referred to as an optical flow. An optical flow may be defined as a set of motion vectors corresponding to samples or blocks in an image.

The optical flow determines how the positions of samples in the previous reconstructed image (y _i-1 ) are changed in the current image (x _i ), or where samples of the current image (x _i ) are located in the previous reconstructed image (y _i-1 ). indicates where it is located. For example, if the sample located at (1, 1) in the current image (x _i ) is located at (2, 1) in the previous reconstructed image (y _i-1 ), the optical flow or motion vector for the sample is ( 1 (=2-1), 0 (=1-1)).

In the process of encoding and decoding an image using AI, the first encoder 110 and the first decoder 150 are used to obtain the current optical flow g _i for the current image x _i .

Specifically, the previous reconstructed image y _i-1 and the current image x _i are input to the first encoder 110 . The first encoder 110 processes the current image (x _i ) and the previous reconstructed image (y _i-1 ) according to parameters set as a result of training, and outputs feature data (w _i ) for the current optical flow.

The feature data w _i for the current optical flow represents a latent feature of the current optical flow.

The feature data w _i for the current optical flow is input to the first decoder 150 . The first decoder 150 processes the input feature data w _i according to parameters set as a result of training and outputs a current optical flow g _i .

The previously reconstructed image y _i-1 is warped 190 according to the current optical flow g _i , and as a result of the warping 190 , the current prediction image x' _i is obtained. The warping 190 is a type of geometric deformation that moves the positions of samples in an image. Warping (190) the previous reconstructed image (y _i-1 ) according to the optical flow (g _i ) indicating the relative positional relationship between samples in the previous reconstructed image (y _i-1 ) and samples in the current image (x _i ) ), a current prediction image (x' _i ) similar to the current image (x _i ) is obtained. For example, if the sample located at (1, 1) in the previous reconstructed image (y _i-1 ) is most similar to the sample located at (2, 1) in the current image (x _i ), warping 190 is performed. The position of the sample located at (1, 1) in the previous reconstructed image y _i-1 may be changed to (2, 1).

Since the current prediction image (x' _i ) generated from the previously restored image (y _i-1 ) is not the current image (x _i ) itself, the difference between the current prediction image (x' _i ) and the current image (x _i ) Current residual image data r _i corresponding to may be obtained.

As an example, current residual image data r _i may be obtained by subtracting sample values in the current prediction image x' _i from sample values in the current image x _i .

The current residual image data r _i is input to the second encoder 130 . The second encoder 130 processes the current residual image data (r _i ) according to parameters set as a result of training and outputs feature data ( _vi ) for the current residual image data.

The feature data ( _vi ) for the current residual image data is input to the second decoder 170 . The second decoder 170 processes the input feature data v _i according to parameters set as a result of training and outputs current residual image data r' _i .

The current reconstructed image (y _i ) is obtained by combining the current prediction image (x' _i ) generated through the warping 190 on the previous reconstructed image (y _i-1 ) and the current residual image data (r' _i ). is obtained

In the inter prediction process shown in FIG. 1 , the feature data w _i for the current optical flow obtained through the first encoder 110 is input to the first decoder 150 .

When the encoding and decoding process of the current image (x _i ) is viewed from the viewpoint of the encoding apparatus, the encoding apparatus signals the characteristic data (w _{i )} of the current optical flow to the decoding apparatus. ) to generate the corresponding bitstream. However, when the motion of the object included in the current image (x _i ) and the previous image (x _i-1 ) is large, the size of the sample values included in the current optical flow is large. The bit rate of the bitstream generated based on the feature data w _i may also be increased.

In the embodiments described below, the size of a bitstream generated as a result of encoding for the current optical flow is reduced by using the previous optical flow. The correlation between the previous optical flow and the current optical flow will be described with reference to FIG. 2 .

Referring to FIG. 2 , the first optical flow 25 is obtained between the current image 23 and the first previous image 22 , and the second optical flow 25 is obtained between the first previous image 22 and the second previous image 21 . 2 optical flows 24 are obtained.

The first optical flow 25 and the second optical flow 24 shown in FIG. 2 are visualized according to magnitudes of samples included in the optical flow or magnitudes of motion vectors.

The first optical flow 25 may be referred to as a current optical flow, and the second optical flow 24 may be referred to as a previous optical flow.

Referring to FIG. 2 , the similarity between the first optical flow 25 and the second optical flow 24 may be confirmed. For example, it is possible to see the similarity of sample values in area A in the first optical flow 25 and area B in the second optical flow 24 .

Since objects in temporally continuous images tend to move linearly, the similarity between the first optical flow 25 and the second optical flow 24 can be predicted.

That is, when the previous optical flow (the second optical flow 24) is used to encode the current optical flow (the first optical flow 25) for the current image 23, the encoding result for the current optical flow is used. The size of the generated bitstream may be reduced.

3 is a diagram illustrating a configuration of an image decoding apparatus 300 according to an exemplary embodiment.

Referring to FIG. 3 , the image decoding apparatus 300 according to an embodiment includes an acquirer 310 and a prediction decoder 330 .

The acquisition unit 310 and the prediction decoding unit 330 may be implemented as a processor, and the acquisition unit 310 and the prediction decoding unit 330 may operate according to instructions stored in a memory (not shown).

Although FIG. 3 shows the acquisition unit 310 and the prediction decoding unit 330 separately, the acquisition unit 310 and the prediction decoding unit 330 may be implemented through one processor. In this case, the acquisition unit 310 and the prediction decoding unit 330 may be implemented as a dedicated processor, and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphic processing unit (GPU) and software. It may be implemented through a combination. In addition, the dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processing unit for using an external memory.

The acquisition unit 310 and the prediction decoding unit 330 may include a plurality of processors. In this case, it may be implemented as a combination of dedicated processors, or may be implemented through a combination of software and a plurality of general-purpose processors such as an AP, CPU, or GPU.

The acquisition unit 310 acquires a bitstream including an encoding result of the current image.

The acquisition unit 310 may receive a bitstream from an image encoding apparatus 1200, which will be described later, through a network. In one embodiment, the acquisition unit 310 is a hard disk, a magnetic medium such as a floppy disk and magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk (floptical disk) -optical medium), etc., may be obtained from a data storage medium including a bitstream.

The obtainer 310 parses the bitstream to obtain feature data of the current residual optical flow and feature data of the current residual image data.

The current residual optical flow corresponds to the difference between the current predicted optical flow and the current optical flow predicted from the previous optical flow, and the current residual image data is based on the difference between the current predicted image and the current image predicted from the previous reconstructed image. respond

The feature data of the current residual optical flow and the feature data of the current residual image data may be obtained as a result of processing by a neural network-based encoder.

In an embodiment, the obtaining unit 310 obtains a first bitstream corresponding to the feature data of the current residual optical flow and a second bitstream corresponding to the feature data of the current residual image data, and the first bitstream and parsing the second bitstream, respectively, to obtain feature data of the current residual optical flow and feature data of the current residual image data.

The feature data of the current residual optical flow and the feature data of the current residual image data are transmitted to the predictive decoding unit 330 , and the predictive decoding unit 330 is configured to convert the feature data of the current residual optical flow and the current residual image data. A current reconstructed image corresponding to the current image is obtained using the feature data.

Depending on the implementation, the bitstream may not include feature data of the current residual image data. The obtainer 310 may obtain feature data of the current residual optical flow from the bitstream, and the predictive decoder 330 may reconstruct the current optical flow. In this case, the image decoding apparatus 300 may be referred to as an optical flow decoding apparatus.

The current optical flow reconstructed by the predictive decoder 330 may be transmitted to another device, and a current reconstructed image may be generated by the other device based on the current optical flow.

Specifically, the other device may generate the current reconstructed image by combining the current residual image data obtained from the bitstream and the current prediction image generated from the previous reconstructed image according to the current optical flow.

Hereinafter, operations of the acquisition unit 310 and the prediction decoding unit 330 will be described in detail with reference to FIGS. 4 and 5 .

FIG. 4 is a diagram illustrating the configuration of the acquisition unit 310 shown in FIG. 3 .

Referring to FIG. 4 , the acquisition unit 310 includes an entropy decoding unit 311 and an inverse quantization unit 313 .

The entropy decoder 311 entropy-codes bins included in the bitstream to obtain quantized feature data of the current residual optical flow and quantized feature data of the current residual image data.

The inverse quantizer 313 inversely quantizes the quantized feature data of the current residual optical flow and the quantized feature data of the current residual image data, respectively, to obtain the feature data of the current residual optical flow and the feature data of the current residual image data. to acquire

According to an embodiment, the obtaining unit 310 may further include an inverse transform unit. The inverse transform unit inversely transforms the feature data output from the inverse quantization unit 313 from the frequency domain to the spatial domain. When the image encoding apparatus 1200, which will be described later, transforms the feature data of the current residual optical flow and the feature data of the current residual image data from the spatial domain to the frequency domain, the inverse transform unit is the feature data output from the inverse quantizer 313 . can be inversely transformed from the frequency domain to the spatial domain.

Also, depending on the implementation, the acquisition unit 310 may not include the inverse quantization unit 313 . That is, feature data of the current residual optical flow and feature data of the current residual image data may be obtained through processing by the entropy decoder 311 .

Also, according to an embodiment, the obtainer 310 may acquire feature data of the current residual optical flow and feature data of the current residual image data by performing only inverse binarization on bins included in the bitstream. This is when the image encoding apparatus 1200 generates a bitstream by binarizing the feature data of the current residual optical flow and the feature data of the current residual image data, that is, when the image encoding apparatus 1200 generates the current residual optical flow. This is for a case in which entropy encoding, transformation, and quantization are not applied to the feature data of the flow and the feature data of the current residual image data.

Next, FIG. 5 is a diagram illustrating the configuration of the prediction decoding unit 330 shown in FIG. 3 .

Referring to FIG. 5 , the predictive decoder 330 includes a first decoder 331 , a second decoder 333 , an optical flow predictor 334 , a first combiner 336 , a motion compensator 335 and A second coupling part 337 may be included.

The first decoder 331 and the second decoder 333 may be stored in a memory. In one embodiment, the first decoder 331 and the second decoder 333 may be implemented as at least one dedicated processor for AI.

The feature data of the current residual optical flow output by the acquirer 310 is input to the first decoder 331 , and feature data of the current residual image data is input to the second decoder 333 .

According to an embodiment, in order to accurately reconstruct the current residual image data, after the feature data of the current residual optical flow or the feature data of the current optical flow is concatenated with the feature data of the current residual image data, the second decoder (333) may also be input. Here, concatenation may refer to a process of combining two or more feature data in a channel direction.

The first decoder 331 obtains the current residual optical flow by processing the feature data of the current residual optical flow according to parameters set through training. The current residual optical flow is one-dimensional or two-dimensional data, and may consist of a plurality of samples.

The second decoder 333 obtains current residual image data by processing the feature data of the current residual image data according to parameters set through training. The current residual image data is one-dimensional or two-dimensional data, and may include a plurality of samples.

The optical flow prediction unit 334 obtains the current predicted optical flow by using at least one of the previous optical flow, the characteristic data of the previous optical flow, and the characteristic data of the previous residual optical flow.

The current prediction optical flow is one-dimensional or two-dimensional data, and may consist of a plurality of samples.

In an embodiment, the optical flow prediction unit 334 may determine the previous optical flow as the current predicted optical flow.

As described with reference to FIG. 2 , due to the linear motion of an object in successive images, it is highly likely that the previous optical flow is very similar to the current optical flow. Accordingly, when the previous optical flow is determined as the current prediction optical flow, the sizes of the sample values of the current residual optical flow and the sample values of the feature data of the current residual optical flow may be reduced.

The current prediction optical flow obtained by the optical flow prediction unit 334 and the current residual optical flow obtained through the first decoder 331 are provided to the first combining unit 336 .

The first combiner 336 reconstructs the current optical flow by combining the current prediction optical flow and the current residual optical flow. The first combiner 336 may reconstruct the current optical flow by summing sample values of the current prediction optical flow and sample values of the current residual optical flow.

The motion compensator 335 generates a current prediction image similar to the current image by processing the previously reconstructed image according to the current optical flow. The previously reconstructed image is an image reconstructed through decoding on a previous image that was a processing target before the current image is processed.

The motion compensator 335 may warp the previously reconstructed image according to the current optical flow to generate the current prediction image. Warping for generation of a current prediction image is an example, and the motion compensator 335 performs various image processing for changing positions of samples in a previously restored image in order to generate a current prediction image similar to the current image. can be applied to

The current prediction image generated by the motion compensator 335 is provided to the second combiner 337 .

The second combiner 337 obtains a current reconstructed image by combining the current prediction image and the current residual image data. In an example, the second combiner 337 may obtain a current reconstructed image including sum values of sample values of the current prediction image and sample values of the current residual image data.

The current reconstructed image and the current optical flow may be used in the decoding process of the next image.

According to an embodiment, the prediction decoding unit 330 may reconstruct the current optical flow from feature data of the current residual optical flow and provide the reconstructed current optical flow to another device. In this case, the second decoder 333 , the motion compensator 335 , and the second combiner 337 may not be included in the prediction decoder 330 .

According to an embodiment, when the current residual image data can be obtained from the bitstream, the second decoder 333 may not be included in the prediction decoder 330 . That is, the prediction decoding unit 330 may generate a current reconstructed image by combining the current residual image data obtained by the obtaining unit 310 from the bitstream with the current prediction image.

According to an embodiment of the present disclosure, since the bitstream is generated based on the current residual optical flow including samples having a size smaller than that of the current optical flow, the bitrate is lower than when the bitstream is generated from the current optical flow. is possible to achieve

Previously, an embodiment in which the optical flow prediction unit 334 determines the previous optical flow as the current prediction optical flow has been described. Hereinafter, another embodiment of the optical flow prediction unit 334 with reference to FIGS. 6 to 8 is described. An operation according to an example will be described.

6 is a diagram illustrating a configuration of an optical flow prediction unit 600 according to an embodiment.

Referring to FIG. 6 , the optical flow prediction unit 600 includes a first prediction neural network 610 . The first prediction neural network 610 may be stored in a memory. In an embodiment, the first prediction neural network 610 may be implemented as at least one dedicated processor for AI.

At least one of the previous optical flow, feature data of the previous optical flow, and feature data of the previous residual optical flow is input to the first prediction neural network 610 to obtain the current predicted optical flow.

The feature data of the previous optical flow represents the latent features of the previous optical flow used during the restoration process of the previous image.

In an embodiment, when the previous image is a P (predictive) frame after the I (intra) frame, the feature data of the previous optical flow may be acquired during the restoration process of the previous optical flow. The I frame and the P frame will be described later.

In another embodiment, the predictive decoder 330 may obtain feature data of the previous optical flow by applying the restored previous optical flow to the neural network after the previous optical flow is restored.

The first prediction neural network 610 processes at least one of the characteristic data of the previous optical flow, the characteristic data of the previous optical flow, and the characteristic data of the previous residual optical flow through the parameters set through training to obtain the current prediction optical flow.

As described with reference to FIG. 5 , by combining the current prediction optical flow and the current residual optical flow, the current optical flow used to generate the current prediction image is obtained. As will be described later with reference to FIGS. 20 and 21 , the first prediction neural network 610 includes the first encoder 1211 , the second encoder 1215 , the first encoder 1211 , the second encoder 1215 , and the It is trained with a first decoder 331 and a second decoder 333 .

Since the data output by the first prediction neural network 610 is combined with the current residual optical flow output by the first decoder 331 and used to generate the current prediction image, the first prediction neural network 610 is It can be trained to output the difference between the current optical flow and the current residual optical flow, that is, the current predicted optical flow.

7 is a diagram illustrating a configuration of an optical flow prediction unit 700 according to another embodiment.

Referring to FIG. 7 , the optical flow prediction unit 700 includes a second prediction neural network 710 and a change unit 720 .

The second prediction neural network 710 may be stored in a memory. In an embodiment, the second prediction neural network 710 may be implemented as at least one dedicated processor for AI.

At least one of the previous optical flow, feature data of the previous optical flow, and feature data of the previous residual optical flow is input to the second prediction neural network 710 .

The second prediction neural network 710 acquires a second-order optical flow between the current prediction optical flow and the previous optical flow through a parameter set through training.

The second-order optical flow is an optical flow between optical flows, and may be defined as a set of motion vectors corresponding to samples or blocks in the optical flow.

The second-order optical flow indicates how the positions of samples in the previous optical flow have changed in the current prediction optical flow or where reference samples of the samples of the current prediction optical flow are located in the previous optical flow. For example, if the sample located in (1, 1) in the previous optical flow is located in (2, 1) in the current prediction optical flow, the second-order optical flow or motion vector for the sample is (1(=2) -1), 0 (=1-1)).

The change unit 720 obtains a current predicted optical flow by processing the previous optical flow according to the second-order optical flow.

The operation of the changer 720 is similar to that of the motion compensator 335 illustrated in FIG. 5 . That is, the motion compensator 335 warps the previous reconstructed image according to the current optical flow to obtain a current predicted image, and the changer 720 warps the previous optical flow according to the second-order optical flow to obtain the current prediction image. A predictive optical flow can be obtained.

Warping for generation of the current prediction optical flow is an example, and the change unit 720 transfers various processes of changing the positions of samples in the previous optical flow to generate a current prediction optical flow similar to the current optical flow. It can be applied to optical flow.

Since the data output by the second prediction neural network 710 is used to change the positions of samples in the previous optical flow, the second prediction neural network 710 through training of the second prediction neural network 710 based on the loss information is Data for changing the previous optical flow to the current predicted optical flow, that is, the second-order optical flow may be output.

8 is a diagram illustrating a configuration of an optical flow prediction unit 800 according to another embodiment.

Referring to FIG. 8 , the optical flow prediction unit 800 includes a third decoder 810 and a change unit 720 . The third decoder 810 may be stored in a memory. In one embodiment, the third decoder 810 may be implemented as at least one dedicated processor for AI.

The third decoder 810 acquires the second-order optical flow by processing feature data of the second-order optical flow according to parameters set through training.

The feature data of the second-order optical flow may be obtained from a bitstream. The obtainer 310 may obtain feature data of the second-order optical flow from the bitstream and provide it to the prediction decoder 330 .

The image encoding apparatus 1200 may generate a bitstream including feature data of the current residual optical flow and feature data of the current residual image data. A bitstream further including optical flow feature data may be generated. This will be described later with reference to FIG. 14 .

The change unit 720 may obtain a current predicted optical flow by processing a previous optical flow according to the second-order optical flow.

In an embodiment, the changer 720 may warp the previous optical flow according to the second-order optical flow to obtain the current predicted optical flow. Warping for generation of the current prediction optical flow is an example, and the change unit 720 sets the positions of samples in the previous optical flow to generate a current prediction optical flow similar to the current optical flow as a second-order optical flow. Various processing that changes according to the previous optical flow can be applied.

In the embodiment shown in FIG. 8 , the feature data of the second-order optical flow provided from the image encoding apparatus 1200 is input to the third decoder 810 and processed. Therefore, the first prediction neural network 610 and the second prediction neural network 710 that receive and process at least one of the feature data of the previous optical flow, the previous optical flow, and the feature data of the previous residual optical flow are processed by the third decoder ( 810) can be reduced. Because the third decoder 810 processes the feature data of the second-order optical flow indicating the characteristics of the second-order optical flow itself, the first prediction neural network 610 and the second prediction neural network 710 are the current This is because a previous optical flow, characteristic data of a previous optical flow, and/or characteristic data of a previous residual optical flow, which has relatively low relevance to the predicted optical flow and the second-order optical flow, is processed.

9 is a flowchart of a method of restoring an optical flow according to an embodiment.

In operation S910 , the image decoding apparatus 300 obtains feature data for the current residual optical flow from the bitstream for the current image.

The image decoding apparatus 300 may obtain feature data for the current residual optical flow by applying at least one of inverse binarization, entropy decoding, inverse quantization, and inverse transform to bins included in the bitstream.

In operation S920, the image decoding apparatus 300 obtains the current residual optical flow by applying the feature data of the current residual optical flow to the first decoder based on the neural network.

In operation S930, the image decoding apparatus 300 obtains a current prediction optical flow by using at least one of the previous optical flow, feature data for the previous optical flow, and feature data for the previous residual optical flow.

In an embodiment, the image decoding apparatus 300 may determine a previous optical flow as a current prediction optical flow.

In another embodiment, the image decoding apparatus 300 applies at least one of the feature data for the previous optical flow, the feature data for the previous optical flow, and the feature data for the previous residual optical flow to the first prediction neural network 610 to predict the current Optical flow can be obtained.

In another embodiment, the image decoding apparatus 300 applies at least one of the feature data for the previous optical flow, the feature data for the previous optical flow, and the feature data for the previous residual optical flow to the second prediction neural network 710 so that the second- An order optical flow may be obtained, and a previous optical flow may be processed according to the second-order optical flow to obtain a current predicted optical flow.

In another embodiment, the image decoding apparatus 300 obtains a second-order optical flow by applying feature data for a second-order optical flow obtained from a bitstream to the third decoder 810, and uses the previous optical flow The current predicted optical flow may be obtained by processing according to the second-order optical flow.

In operation S940, the image decoding apparatus 300 reconstructs the current optical flow by using the current residual optical flow and the current prediction optical flow. The image decoding apparatus 300 may acquire the current optical flow by summing sample values of the current residual optical flow and sample values of the current prediction optical flow.

In an embodiment, the image decoding apparatus 300 obtains feature data for the current residual image data from the bitstream, and applies the feature data for the current residual image data to the second decoder 333 to obtain the current residual image data. Image data can be acquired. In addition, the image decoding apparatus 300 may obtain a current predicted image by processing the previous reconstructed image according to the current optical flow, and may obtain a current reconstructed image by combining the current predicted image and current residual image data.

In another embodiment, the image decoding apparatus 300 may obtain current residual image data from a bitstream. In addition, the image decoding apparatus 300 may obtain a current predicted image by processing the previous reconstructed image according to the current optical flow, and may obtain a current reconstructed image by combining the current predicted image and current residual image data.

In another embodiment, the image decoding apparatus 300 may provide the current optical flow to another device so that the current reconstructed image is acquired by the other device.

Meanwhile, the inter prediction process described with reference to FIGS. 3 to 9 considers a case in which a previous image is processed through inter prediction. This is because the previous optical flow used to reconstruct the current optical flow is generated in the inter prediction process of the previous image.

That is, the inter prediction process described with reference to FIGS. 3 to 9 is applied when the current image corresponds to a P frame following a P (predictive) frame, that is, when the previous image is a P frame and the current image is a P frame. can Here, the P frame means an image or frame that can be reconstructed through intra prediction or inter prediction. An image or frame that can be reconstructed only through intra prediction is referred to as an I (intra) frame.

Therefore, if the previous image is an I frame, the previous optical flow is not obtained, so the case where the current image is the P frame after the I frame, that is, the previous image is the I frame, and the current image is the P frame The inter prediction process for

10 is a diagram illustrating another configuration of the predictive decoding unit 330 .

Referring to FIG. 10 , the prediction decoding unit 330 includes a fourth decoder 1010 , a second decoder 333 , a motion compensator 335 , and a second combiner 337 .

The fourth decoder 1010 may be stored in a memory. In one embodiment, the fourth decoder 1010 may be implemented as at least one dedicated processor for AI.

The fourth decoder 1010 obtains the current optical flow by processing the feature data of the current optical flow according to parameters set through training.

The feature data of the current optical flow may be obtained from a bitstream. That is, the acquirer 310 may acquire the feature data of the current optical flow by applying at least one of inverse binarization, entropy decoding, inverse quantization, and inverse transformation to bins included in the bitstream.

The second decoder 333 obtains current residual image data by processing the feature data of the current residual image data according to parameters set through training.

The motion compensator 335 processes the previously reconstructed image according to the current optical flow to obtain a current predicted image, and the second combiner 337 combines the current predicted image with the current residual image data to obtain the current reconstructed image. do.

According to an embodiment, the predictive decoder 330 may transmit the current optical flow to another device and cause the current reconstructed image to be obtained by the other device. In this case, the second decoder 333 , the motion compensator 335 , and the second combiner 337 may not be included in the prediction decoder 330 .

Meanwhile, although not shown in FIGS. 5 and 10 , the predictive decoding unit 330 may further include a determining unit that determines whether the current image is a P frame following the I frame or a P frame following the P frame.

When the current image is a P frame after the P frame, the predictive decoding unit 330 performs the current optical When the flow is restored and the current image is the P frame after the I frame, the current optical flow may be restored through the fourth decoder 1010 illustrated in FIG. 10 .

11 is a flowchart of a method of restoring an optical flow according to another embodiment.

In step S1110 , the image decoding apparatus 300 determines whether the current image is the P frame after the I frame.

When the current image is the P frame after the I frame, in operation S1120 , the image decoding apparatus 300 obtains feature data for the current optical flow from the bitstream.

The image decoding apparatus 300 may obtain feature data for the current optical flow by applying at least one of inverse binarization, entropy decoding, inverse quantization, and inverse transformation to bins included in the bitstream.

In operation S1130 , the image decoding apparatus 300 obtains the current optical flow by applying the feature data for the current optical flow to the fourth decoder 1010 .

When the current image is not the P frame after the I frame, that is, when the current image is the P frame after the P frame, the image decoding apparatus 300 performs the current optical flow through steps S910 to S940 shown in FIG. 9 . can be restored.

Hereinafter, an operation of the image encoding apparatus 1200 will be described with reference to FIGS. 12 to 15 .

12 is a diagram illustrating a configuration of an image encoding apparatus 1200 according to an embodiment.

Referring to FIG. 12 , the image encoding apparatus 1200 includes a predictive encoder 1210 , a generator 1230 , an acquirer 1250 , and a predictive decoder 1270 .

The predictive encoder 1210 , the generator 1230 , the acquirer 1250 , and the predictive decoder 1270 may be implemented as a processor, and the predictive encoder 1210 , the generator 1230 , and the acquirer 1250 . ) and the prediction decoding unit 1270 may operate according to instructions stored in a memory (not shown).

12 shows the prediction encoder 1210 , the generator 1230 , the acquirer 1250 , and the predictive decoder 1270 separately, the predictive encoder 1210 , the generator 1230 , and the acquirer 1250 and the prediction decoding unit 1270 may be implemented through one processor. In this case, the prediction encoder 1210 , the generator 1230 , the acquirer 1250 , and the prediction decoder 1270 are implemented as a dedicated processor, or an application processor (AP), central processing unit (CPU), or GPU ( It may be implemented through a combination of a general-purpose processor such as a graphic processing unit and software. In addition, the dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processing unit for using an external memory.

The predictive encoder 1210 , the generator 1230 , the acquirer 1250 , and the predictive decoder 1270 may include a plurality of processors. In this case, it may be implemented as a combination of dedicated processors, or may be implemented through a combination of software and a plurality of general-purpose processors such as an AP, CPU, or GPU.

The prediction encoder 1210 obtains feature data of the current residual optical flow and feature data of the current residual image data by using the current image and the previous reconstructed image.

The prediction encoder 1210 may use a neural network-based first encoder 1211 and a neural network-based second encoder 1215 to obtain feature data of the current residual optical flow and feature data of the current residual image data. have.

The feature data of the current residual optical flow obtained by the prediction encoder 1210 and the feature data of the current residual image data are transmitted to the generator 1230 .

The generator 1230 generates a bitstream from feature data of the current residual optical flow and feature data of the current residual image data. In an embodiment, the generator 1230 may generate a first bitstream corresponding to the feature data of the current residual optical flow and a second bitstream corresponding to the feature data of the current residual image data.

The bitstream may be transmitted from the image decoding apparatus 300 through a network. Further, in one embodiment, the bitstream is a hard disk, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks. It may be recorded on a data storage medium including an optical medium).

The acquirer 1250 acquires feature data of the current residual optical flow and feature data of the current residual image data from the bitstream generated by the generator 1230 . According to an embodiment, the obtainer 1250 may receive feature data of the current residual optical flow and feature data of the current residual image data from the prediction encoder 1210 .

The feature data of the current residual optical flow and the feature data of the current residual image data are transmitted to the predictive decoder 1270, and the predictive decoder 1270 uses the feature data of the current residual optical flow to generate the current optical flow. reconstructed, and a current reconstructed image is obtained using the feature data of the current optical flow and the current residual image data.

The current optical flow and the current reconstructed image obtained by the prediction decoder 1270 may be used in the encoding process of the next image.

The configuration and operation of the acquisition unit 1250 and the prediction decoding unit 1270 are the same as the operations of the acquisition unit 310 and the prediction decoding unit 330 illustrated in FIGS. 3 to 5 , and thus a detailed description thereof will be omitted. However, as shown in FIG. 13 , since the predictive encoder 1210 includes the optical flow predictor 1217 , unlike the predictive decoder 330 illustrated in FIG. 5 , the predictive decoder 1270 uses the optical flow The prediction unit 334 may not be included. This is because the prediction decoding unit 1270 may use the current prediction optical flow obtained by the optical flow prediction unit 1217 included in the prediction encoding unit 1210 .

In an embodiment, the prediction encoder 1210 obtains feature data of the current residual optical flow by using the current image and the previously reconstructed image, and the generator 1230 corresponds to the feature data of the current residual optical flow. You can create a bitstream. In addition, the obtainer 1250 may obtain feature data of the current residual optical flow from the bitstream, and the predictive decoder 1270 may reconstruct the current optical flow based on the feature data of the current residual optical flow.

That is, since the current optical flow is encoded through the prediction encoder 1210 , the generator 1230 , the acquirer 1250 , and the predictive decoder 1270 , in this case, the image encoding apparatus 1200 is the optical flow encoding apparatus may be referred to as

The current optical flow reconstructed by the predictive decoder 1270 may be transmitted to another device, and the current residual image data may be encoded by the other device. Specifically, the other device may encode the current residual image data corresponding to the difference between the current prediction image and the current image obtained from the previous reconstructed image according to the current optical flow.

The configuration of the prediction encoder 1210 and the generator 1230 will be described in more detail with reference to FIGS. 13 to 15 .

FIG. 13 is a diagram showing the configuration of the predictive encoder 1210 shown in FIG. 12 .

The predictive encoder 1210 includes an optical flow predictor 1217 , a first encoder 1211 , a second encoder 1215 , and a subtractor 1213 .

The first encoder 1211 and the second encoder 1215 may be stored in a memory. In one embodiment, the first encoder 1211 and the second encoder 1215 may be implemented as at least one dedicated processor for AI.

Referring to FIG. 13 , the optical flow prediction unit 1217 obtains a current predicted optical flow by using at least one of a previous optical flow, feature data of a previous optical flow, and feature data of a previous residual optical flow.

The optical flow prediction unit 1217 may acquire the current prediction optical flow in the same manner as the optical flow prediction unit 334 of the image decoding apparatus 300 .

As an example, the optical flow prediction unit 1217 may obtain the current prediction optical flow by having the same configuration as the optical flow prediction unit 600 or 700 illustrated in FIG. 6 or 7 .

Specifically, as described with reference to FIG. 6 , the optical flow prediction unit 1217 converts at least one of the feature data for the previous optical flow, the previous optical flow, and the previous residual optical flow to the first prediction neural network ( 610) to obtain a current predicted optical flow.

In addition, as described with reference to FIG. 7 , the optical flow prediction unit 1217 converts at least one of the feature data for the previous optical flow, the feature data for the previous optical flow, and the feature data for the previous residual optical flow to the second prediction neural network 710 . is applied to obtain a second-order optical flow, and a previous optical flow may be processed according to the second-order optical flow to obtain a current predicted optical flow.

As another example, the optical flow prediction unit 1217 may determine the previous optical flow as the current predicted optical flow.

When the optical flow prediction unit 334 of the image decoding apparatus 300 includes the third decoder 810 and the changer 720 as shown in FIG. 8 , the optical flow prediction of the image encoding apparatus 1200 is The configuration of the unit 1217 will be described later with reference to FIG. 14 .

At least one of a current image, a previous reconstructed image, and a current prediction optical flow is input to the first encoder 1211 . At least one of the current image, the previous reconstructed image, and the current prediction optical flow may be concatenated and then input to the first encoder 1211 .

Since information on the current optical flow can be derived from the current image and the previous reconstructed image, the first encoder 1211 generates the current optical flow identified from the current image and the previous reconstructed image, and the optical flow prediction unit 1217 . It is possible to output feature data of the current residual optical flow corresponding to the difference therebetween using the current predicted optical flow.

The first encoder 1211 processes at least one of a current image, a previous reconstructed image, and a current predicted optical flow according to a parameter set as a result of training, and outputs feature data of the current residual optical flow.

The prediction decoding unit 1270 shown in FIG. 12 reconstructs the current optical flow based on the characteristic data of the current residual optical flow, and subtracts the current prediction image generated from the previous reconstructed image according to the current optical flow. is provided as

The subtractor 1213 obtains current residual image data between the current image and the current prediction image. The subtractor 1213 may obtain current residual image data by subtracting sample values of the current prediction image from sample values of the current image.

The current residual image data is input to the second encoder 1215, and the second encoder 1215 processes the current residual image data according to parameters set as a result of training to output feature data of the current residual image data. .

The above-described generator 1230 generates a bitstream based on the feature data of the current residual optical flow output from the prediction encoder 1210 and the feature data of the current residual image data.

FIG. 14 is a diagram showing the configuration of the optical flow prediction unit 1217 of the video encoding apparatus 1200 corresponding to the optical flow prediction unit 800 shown in FIG. 8 .

Referring to FIG. 14 , the optical flow predictor 1217 includes a third encoder 1410 , a third decoder 810 , and a changer 720 . 8 , it can be seen that the optical flow prediction unit 1217 includes a third encoder 1410 .

The third encoder 1410 and the third decoder 810 may be stored in a memory. In an embodiment, the third encoder 1410 and the third decoder 810 may be implemented as at least one dedicated processor for AI.

The third encoder 1410 processes at least one of the current image, the previous reconstructed image, the previous optical flow, the feature data of the previous optical flow, and the feature data of the previous residual optical flow according to the parameters set according to training to process the second-order optical Acquire flow characteristic data.

A bitstream corresponding to the feature data of the second-order optical flow may be provided to the image decoding apparatus 300 .

The third decoder 810 acquires the second-order optical flow by processing feature data of the second-order optical flow according to parameters set through training.

The change unit 720 may obtain a current predicted optical flow by processing a previous optical flow according to the second-order optical flow.

In an embodiment, the changer 720 may warp the previous optical flow according to the second-order optical flow to obtain the current predicted optical flow. Warping is an example, and the change unit 720 may apply various processes for changing positions of samples in the previous optical flow to the previous optical flow to generate the current predicted optical flow.

The optical flow prediction unit 1217 illustrated in FIG. 14 obtains feature data of the second-order optical flow by using various types of data available to the image encoding apparatus 1200 . And, the feature data of the second-order optical flow is signaled to the image decoding apparatus 300 . The optical flow prediction unit 800 of the image decoding apparatus 300 processes the feature data of the second-order optical flow signaled from the image encoding apparatus 1200 with the third decoder 810 to obtain a second-order optical flow. .

The prediction optical flow obtained by the image decoding apparatus 300 using the feature data of the second-order optical flow signaled from the image encoding apparatus 1200 is more accurate than the prediction optical flow obtained by the image decoding apparatus 300 itself. do. This is because the types of data that the image encoding apparatus 1200 can use to obtain feature data of the second-order optical flow are larger than those of the image decoding apparatus 300 . For example, since the image decoding apparatus 300 cannot use the current image before decoding the current image, for example, the optical flow prediction units 600 and 700 shown in FIGS. 6 and 7 may predict the current predicted optical flow. The current image is not used for acquisition.

FIG. 15 is a diagram illustrating the configuration of the generator 1230 shown in FIG. 12 .

Referring to FIG. 15 , the generator 1230 includes a quantizer 1231 and an entropy encoder 1233 .

The quantizer 1231 quantizes the feature data of the current residual optical flow and the feature data of the current residual image data.

The entropy encoder 1233 entropy-codes the quantized feature data of the current residual optical flow and the quantized feature data of the current residual image data to generate a bitstream.

According to an embodiment, the generator 1230 may further include a converter. The transform unit transforms the feature data of the current residual optical flow and the feature data of the current residual image data from the spatial domain to the frequency domain, and provides them to the quantizer 1231 .

Also, depending on the implementation, the generator 1230 may not include the quantizer 1231 . That is, the bitstream corresponding to the feature data of the current residual optical flow and the feature data of the current residual image data may be obtained through processing by the entropy encoder 1233 .

Also, according to an embodiment, the generator 1230 may generate a bitstream by performing binarization on the feature data of the current residual optical flow and the feature data of the current residual image data. That is, when the generator 1230 performs only binarization, the quantizer 1231 and the entropy encoder 1233 may not be included in the generator 1230 .

16 is a flowchart of a method of encoding an optical flow according to an embodiment.

In operation S1610 , the image encoding apparatus 1200 obtains a current prediction optical flow from at least one of the previous optical flow, feature data for the previous optical flow, and feature data for the previous residual optical flow.

In an embodiment, the image encoding apparatus 1200 may determine a previous optical flow as a current prediction optical flow.

In another embodiment, the image encoding apparatus 1200 applies at least one of feature data for a previous optical flow, a previous optical flow, and a previous residual optical flow to the first prediction neural network 610 to predict the current Optical flow can be obtained.

In another embodiment, the image encoding apparatus 1200 applies at least one of the feature data for the previous optical flow, the previous optical flow, and the feature data for the previous residual optical flow to the second prediction neural network 710 so that the second- An order optical flow may be obtained, and a previous optical flow may be processed according to the second-order optical flow to obtain a current predicted optical flow.

In another embodiment, the image encoding apparatus 1200 transmits at least one of feature data for the current image, the previous reconstructed image, the previous optical flow, the feature data for the previous optical flow, and the feature data for the previous residual optical flow to the third encoder 1410 ) to obtain feature data of a second-order optical flow, and apply the feature data of the second-order optical flow to the third decoder 810 to obtain a second-order optical flow. In addition, the image encoding apparatus 1200 may obtain a current predicted optical flow by processing the previous optical flow according to the second-order optical flow.

In step S1620, the image encoding apparatus 1200 applies at least one of a current image, a previously reconstructed image, and a current prediction optical flow to the first encoder 1211 based on a neural network to obtain feature data for the current residual optical flow. do.

In operation S1630, the image encoding apparatus 1200 generates a bitstream corresponding to feature data of the current residual optical flow.

In an embodiment, the bitstream may further include feature data of a second-order optical flow and/or feature data of current residual image data.

In an embodiment, the image encoding apparatus 1200 reconstructs a current optical flow from feature data of the current residual optical flow, and processes a previously reconstructed image based on the reconstructed current optical flow to obtain a current predicted image. In addition, the image encoding apparatus 1200 may obtain feature data of the current residual image data by applying the current residual image data corresponding to the difference between the current prediction image and the current image to the second encoder 1215 . The feature data of the current residual image data may be included in the bitstream.

In another embodiment, the image encoding apparatus 1200 reconstructs a current optical flow from feature data of the current residual optical flow, and processes a previous reconstructed image based on the reconstructed current optical flow to obtain a current predicted image. In addition, current residual image data corresponding to a difference between the current prediction image and the current image may be included in the bitstream.

Meanwhile, the encoding process described with reference to FIGS. 12 to 16 considers a case in which a previous image is processed through inter prediction. This is because the previous optical flow used to encode the current optical flow is generated in the inter prediction process of the previous image.

That is, the encoding process described with reference to FIGS. 12 to 16 may be applied when the current image corresponds to the P frame following the P frame. If the previous image is an I frame, the previous optical flow is not obtained. Hereinafter, an encoding process for a case in which the current image is a P frame after the I frame will be described.

17 is a diagram showing another configuration of the predictive encoding unit 1210 .

Referring to FIG. 17 , the prediction encoder 1210 includes a fourth encoder 1710 , a second encoder 1215 , and a subtractor 1213 .

The fourth encoder 1710 and the second encoder 1215 may be stored in a memory. In one embodiment, the fourth encoder 1710 and the second encoder 1215 may be implemented as at least one dedicated processor for AI.

The fourth encoder 1710 obtains feature data of the current optical flow by processing the current image and the previous reconstructed image according to parameters set according to training.

The prediction decoding unit 1270 shown in FIG. 12 reconstructs a current optical flow based on feature data of the current optical flow, and provides a current prediction image generated from a previous reconstructed image according to the current optical flow to the subtractor 1213. do. The prediction decoder 1270 may use the fourth decoder 1010 shown in FIG. 10 to restore the current optical flow.

The subtractor 1213 obtains current residual image data between the current image and the current prediction image. The subtractor 1213 may obtain current residual image data by subtracting sample values of the current prediction image from sample values of the current image.

The current residual image data is input to the second encoder 1215, and the second encoder 1215 processes the current residual image data according to parameters set as a result of training to output feature data of the current residual image data. .

The generator 1230 generates a bitstream based on the feature data of the current optical flow output from the prediction encoder 1210 and the feature data of the current residual image data.

The bitstream may be transmitted from the image decoding apparatus 300 through a network. Further, in one embodiment, the bitstream is a hard disk, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks. It may be recorded on a data storage medium including an optical medium).

Although not shown in FIGS. 13 and 17 , the prediction encoding unit 1210 may further include a determination unit determining whether the current image is a P frame following the I frame or a P frame following the P frame.

The prediction encoder 1210 obtains feature data of the current residual optical flow through the optical flow predictor 1217 and the first encoder 1211 shown in FIG. 13 when the current image is a P frame after the P frame. and, when the current image is the P frame after the I frame, the feature data of the current optical flow may be obtained through the fourth encoder 1710 illustrated in FIG. 17 .

Meanwhile, as described above, the first encoder 1211 , the second encoder 1215 , the third encoder 1410 , the fourth encoder 1710 , the first decoder 331 , the second decoder 333 , and the third decoder At least one of 810 , the fourth decoder 1010 , the first prediction neural network 610 , and the second prediction neural network 710 may include a convolutional layer.

First encoder 1211 , second encoder 1215 , third encoder 1410 , fourth encoder 1710 , first decoder 331 , second decoder 333 , third decoder 810 , 4 A structure that the decoder 1010 , the first prediction neural network 610 , and the second prediction neural network 710 may have will be described with reference to FIG. 18 .

18 is a diagram illustrating a structure of a neural network 1800 according to an embodiment.

18 , input data 1805 is input to a first convolutional layer 1810 . Here, the input data 1805 is the first encoder 1211 , the second encoder 1215 , the third encoder 1410 , the fourth encoder 1710 , the first decoder 331 , and the second It depends on which one of the decoder 333 , the third decoder 810 , the fourth decoder 1010 , the first prediction neural network 610 , and the second prediction neural network 710 .

For example, when the neural network 1800 is the first encoder 1211, the input data 1805 corresponds to a result in which the current image, the previous reconstructed image, and the predicted optical flow are concatenated, and the neural network 1800 is the second In the case of the second encoder 1215 , the input data 1805 may correspond to current residual image data.

3X3X4 displayed in the first convolutional layer 1810 shown in FIG. 18 exemplifies convolution processing on one input data 1805 using four filter kernels having a size of 3x3. As a result of the convolution process, four feature maps are generated by four filter kernels.

The feature maps generated by the first convolutional layer 1810 represent unique properties of the input data 1805 . For example, each feature map may indicate a vertical direction characteristic, a horizontal direction characteristic, or an edge characteristic of the input data 1805 .

A convolution operation in the first convolution layer 1810 will be described in detail with reference to FIG. 19 .

One feature map ( 1950) can be created. Since four filter kernels 1930 are used in the first convolution layer 1810 , four feature maps 1950 may be generated through a convolution operation process using the four filter kernels 1930 .

In FIG. 19 , I1 to I49 indicated in the input data 1805 indicate samples of the input data 1805 , and F1 to F9 indicated in the filter kernel 1930 are samples of the filter kernel 1930 (it may be referred to as a parameter). indicate the Also, M1 to M9 displayed in the feature map 1950 represent samples of the feature map 1950 .

In the convolution operation process, each of the sample values of I1, I2, I3, I8, I9, I10, I15, I16, and I17 of the input data 1805 and F1, F2, F3, F4, F5, Each multiplication operation of F6, F7, F8, and F9 may be performed, and a value obtained by combining (eg, addition operation) result values of the multiplication operation may be assigned as the value of M1 of the feature map 1950 . If the stride of the convolution operation is 2, each of the sample values of I3, I4, I5, I10, I11, I12, I17, I18, I19 of the input data 1805 and F1, F2 of the filter kernel 1930, Each product operation of F3, F4, F5, F6, F7, F8, and F9 may be performed, and a value obtained by combining result values of the product operation may be assigned as the value of M2 of the feature map 1950 .

A convolution operation between sample values in the input data 1805 and samples of the filter kernel 1930 is performed while the filter kernel 1930 moves along the stride until the last sample of the input data 1805 is reached. , a feature map 1950 having a predetermined size may be obtained.

According to the present disclosure, through training on the neural network 1800, parameters of the neural network 1800, for example, samples of the filter kernel 1930 used in convolutional layers of the neural network 1800 (for example, , values of F1, F2, F3, F4, F5, F6, F7, F8, and F9 of the filter kernel 1930 may be optimized.

The convolution layers included in the neural network 1800 may be processed according to the convolution operation process described in relation to FIG. 19 , but the convolution operation process described in FIG. 19 is only an example, and is not limited thereto.

Referring back to FIG. 18 , the feature maps of the first convolutional layer 1810 are input to the first activation layer 1820 .

The first activation layer 1820 may provide a non-linear characteristic to each feature map. The first activation layer 1820 may include, but is not limited to, a sigmoid function, a Tanh function, a Rectified Linear Unit (ReLU) function, and the like.

Giving the nonlinear characteristic to the first activation layer 1820 means changing and outputting some sample values of the feature maps. At this time, the change is performed by applying a non-linear characteristic.

The first activation layer 1820 determines whether to transfer the sample values of the feature map to the second convolution layer 1830 . For example, some sample values of the sample values of the feature map are activated by the first activation layer 1820 and transmitted to the second convolution layer 1830 , and some sample values are activated by the first activation layer 1820 . It is deactivated and is not transmitted to the second convolutional layer 1830 . A unique characteristic of the input data 1805 represented by the feature maps is emphasized by the first activation layer 1820 .

The feature maps 1825 output from the first activation layer 1820 are input to the second convolution layer 1830 . Any one of the feature maps 1825 shown in FIG. 18 is a result of processing the feature map 1950 described with reference to FIG. 19 in the first activation layer 1820 .

3X3X4 displayed in the second convolution layer 1830 exemplifies convolution processing on the input feature maps 1825 using four filter kernels having a size of 3x3. The output of the second convolutional layer 1830 is input to the second activation layer 1840 . The second activation layer 1840 may provide a non-linear characteristic to the input feature maps.

The feature maps 1845 output from the second activation layer 1840 are input to the third convolution layer 1850 . 3X3X1 displayed in the third convolution layer 1850 exemplifies convolution processing to generate one output data 1855 using one filter kernel having a size of 3x3.

The output data 1855 is the neural network 1800 of the first encoder 1211, the second encoder 1215, the third encoder 1410, the fourth encoder 1710, the first decoder 331, the second decoder ( 333 ), the third decoder 810 , the fourth decoder 1010 , the first prediction neural network 610 , and the second prediction neural network 710 .

For example, when the neural network 1800 is the first encoder 1211 , the output data 1855 is feature data for the current residual optical flow, and when the neural network 1800 is the second encoder 1215 , the output data 1855 may be feature data for the current residual image data.

18 shows that the neural network 1800 includes three convolutional layers and two activation layers, but this is only an example, and according to an embodiment, the convolutional layer included in the neural network 1800 and The number of activation layers may be variously changed.

Also, depending on the implementation, the neural network 1800 may be implemented through a recurrent neural network (RNN). In this case, it means changing the CNN structure of the neural network 1800 according to the example of the present disclosure to the RNN structure.

In an embodiment, the image decoding apparatus 300 and the image encoding apparatus 1200 may include at least one arithmetic logic unit (ALU) for the above-described convolution operation and operation of the activation layer.

The ALU may be implemented as a processor. For the convolution operation, the ALU may include a multiplier that performs a multiplication operation between the input data 1805 or the sample values of the feature map output from the previous layer and the sample values of the filter kernel, and an adder that adds the result values of the multiplication. have.

For the calculation of the activation layer, the ALU is a multiplier that multiplies an input sample value by a weight used in a predetermined sigmoid function, a Tanh function, or a ReLU function, and compares the multiplication result with a predetermined value to convert the input sample value to the next layer It may include a comparator that determines whether to transfer to .

Hereinafter, a training method of neural networks used in an image encoding and decoding process will be described with reference to FIGS. 20 to 22 .

20 is a first decoder 331 , a second decoder 333 , a first encoder 1211 , a second encoder 1215 , and a neural network used in an optical flow prediction process 2090 (eg, FIG. 6 ). To explain the training method of the first prediction neural network 610 shown in Fig. 7, the second prediction neural network 710 shown in Fig. 7, or the third decoder 810 and the third encoder 1410 shown in Fig. 14) is a drawing for

In FIG. 20 , a current training image 2010 , a previous reconstructed training image 2030 , and a current reconstructed training image 2050 correspond to the aforementioned current image, the previous reconstructed image, and the current reconstructed image, respectively.

In training the neural network used in the first decoder 331 , the second decoder 333 , the first encoder 1211 , the second encoder 1215 , and the optical flow prediction process 2090 , the current reconstruction training image It should be considered how similar 2050 is to the current training image 2010 and how large the bitrate of a bitstream generated through encoding of the current training image 2010 is. To this end, in one embodiment, the first loss information 2060 corresponding to the similarity between the current training image 2010 and the current reconstruction training image 2050, and the second corresponding to how large the size of the bitstream is The first decoder 331 , the second decoder 333 , the first encoder 1211 , the second encoder 1215 and the optical flow prediction process 2090 according to the loss information 2070 and the third loss information 2080 ) can be trained.

Referring to FIG. 20 , a current predicted optical flow is obtained through an optical flow prediction process 2090 . The current prediction optical flow may be obtained according to the embodiment described with reference to FIG. 6 , the embodiment described with reference to FIG. 7 , or the embodiment described with reference to FIG. 14 . Depending on the implementation, the previous optical flow may be determined as the current prediction optical flow.

The current prediction optical flow, the current training image 2010 and the previous reconstructed training image 2030 are input to the first encoder 1211 . The first encoder 1211 processes the current prediction optical flow, the current training image 2010, and the previous reconstructed training image 2030 to output feature data h _i of the current residual optical flow.

The feature data h _i of the current residual optical flow is input to the first decoder 331 , and the first decoder 331 processes the feature data h _i of the current residual optical flow to process the current residual optical flow. (d _i ) is printed.

The current prediction optical flow and the current residual optical flow d _i are combined 2095 to obtain the current optical flow g _i .

The previous reconstruction training image 2030 is warped 190 according to the current optical flow (g _i ) to generate a current prediction training image (x' _i ), and the current prediction training image (x' _i ) and the current training Current residual image data r _i corresponding to the difference between images 2010 is obtained.

The current residual image data (r _i ) is input to the second encoder 1215 , and the second encoder 1215 processes the current residual image data (ri ) to obtain feature data ( _{vi )} of the current residual image data. ) is output.

The feature data ( _vi ) of the current residual image data is input to the second decoder 333 .

The second decoder 333 processes the feature data ( _{vi) of the current residual image data to output the current residual image data (r' i )} , and the current prediction training image (x' _i ) and the current residual The current reconstruction training image 2050 is obtained by summing the image data r′ _i .

For training the neural network used in the first decoder 331, the second decoder 333, the first encoder 1211, the second encoder 1215, and the optical flow prediction process 2090, the first loss information ( 2060 ), at least one of the second loss information 2070 , and the third loss information 2080 may be obtained.

The first loss information 2060 corresponds to a difference between the current training image 2010 and the current reconstruction training image 2050 . The difference between the current training image 2010 and the current reconstruction training image 2050 is the L1-norm value, the L2-norm value, and the SSIM (Structural) value between the current training image 2010 and the current reconstruction training image 2050 At least one of a similarity) value, a Peak Signal-To-Noise Ratio-Human Vision System (PSNR-HVS) value, a Multiscale SSIM (MS-SSIM) value, a Variance Inflation Factor (VIF) value, and a Video Multimethod Assessment Fusion (VMAF) value. may include

Since the first loss information 2060 is related to the quality of the current reconstruction training image 2050 , the first loss information 2060 may be referred to as quality loss information.

The second loss information 2070 corresponds to the entropy of the feature data h _i of the current residual optical flow or the bit rate of the bitstream corresponding to the feature data h _i of the current residual optical flow. In addition, the third loss information 2080 corresponds to the entropy of the feature data ( _vi ) of the current residual image data or the bit rate of the bitstream corresponding to the feature data ( _vi ) of the current residual image data.

If the bitstream includes both the feature data (h _i ) of the current residual optical flow and the feature data ( _vi ) of the current residual image data, fourth loss information corresponding to the bitrate of the corresponding bitstream can be calculated. have. In this case, the second loss information 2070 and the third loss information 2080 may not be used for training.

Since the second loss information 2070 and the third loss information 2080 are related to the encoding efficiency of the current training image 2010, the second loss information 2070 and the third loss information 2080 have compression loss. Information can be referenced.

The first decoder 331 , the second decoder 333 , the first encoder 1211 , the second encoder 1215 , and the neural network used in the optical flow prediction process 2090 includes the first loss information 2060 , the first The final loss information derived from at least one of the second loss information 2070 and the third loss information 2080 is trained to be reduced or minimized.

Specifically, the first decoder 331 , the second decoder 333 , the first encoder 1211 , the second encoder 1215 , and the neural network used in the optical flow prediction process 2090 determine the value of the preset parameter. Change so that the final loss information is reduced or minimized.

In an embodiment, the final loss information may be calculated according to Equation 1 below.

[Equation 1]

Final loss information = a*First loss information+b*Second loss information+c*Third loss information

In Equation 1, a, b, and c are weights applied to each of the first loss information 2060 , the second loss information 2070 , and the third loss information 2080 , respectively.

According to Equation 1, the neural network used in the first decoder 331 , the second decoder 333 , the first encoder 1211 , the second encoder 1215 and the optical flow prediction process 2090 is the current reconstruction It can be seen that the training image 2050 is as similar to the current training image 2010 as possible and trained in a direction in which the size of the bitstream corresponding to the data output from the first encoder 1211 and the second encoder 1215 is minimized. have.

On the other hand, the training process shown in FIG. 20 corresponds to the training process for the P frame following the P frame. For the case of the P frame following the I frame, the fourth encoder 1710 shown in FIGS. 10 and 17 and Training of the fourth decoder 1010 is required. To this end, the first encoder 1211 and the first decoder 331 shown in FIG. 20 may be replaced with the fourth encoder 1710 and the fourth decoder 1010 . The optical flow prediction process 2090 and the combining process 2095 may not be performed during the training process.

If the training process of the second encoder 1215 , the second decoder 333 , the fourth encoder 1710 , and the fourth decoder 1010 for the case of the P frame following the I frame will be described, the current training image 2010 and the previous reconstruction training image 2030 are input to the fourth encoder 1710 .

The fourth encoder 1710 processes the current training image 2010 and the previous reconstructed training image 2030 to output feature data of the current optical flow, and the feature data of the current optical flow is input to the fourth decoder 1010 . do.

The fourth decoder 1010 processes the feature data of the current optical flow and outputs the current optical flow.

The previous reconstruction training image 2030 is warped 190 according to the current optical flow to generate a current prediction training image (x' _i ), and the current prediction training image (x' _i ) and the current training image (2010) The current residual image data r _i corresponding to the difference between them is obtained.

The current residual image data (r _i ) is input to the second encoder 1215 , and the second encoder 1215 processes the current residual image data (ri ) to obtain feature data ( _{vi )} of the current residual image data. ) is output.

The feature data ( _vi ) of the current residual image data is input to the second decoder 333 . The second decoder 333 processes the feature data ( _{vi) of the current residual image data to output the current residual image data (r' i )} , and the current prediction training image (x' _i ) and the current residual The current reconstruction training image 2050 is obtained by summing the image data r′ _i .

The second encoder 1215 , the second decoder 333 , the fourth encoder 1710 and the fourth decoder 1010 include the first loss information 2060 , the second loss information 2070 , and the third loss information 2080 . ) may be trained to reduce or minimize the final loss information calculated from at least one of.

Here, the first loss information 2060 may correspond to a difference between the current training image 2010 and the current reconstruction training image 2050 . The second loss information 2070 may correspond to the entropy of the feature data of the current optical flow or the bit rate of the bitstream corresponding to the feature data of the current optical flow. Also, the third loss information 2080 may correspond to the entropy of the feature data ( _vi ) of the current residual image data or the bitrate of the bitstream corresponding to the feature data ( _vi ) of the current residual image data.

It can be seen that the second encoder 1215 and the second decoder 333 are used in both the training process for the P frame following the P frame and the training process for the P frame following the I frame.

In one embodiment, the second encoder 1215 and the second decoder 333 are trained through the training process for the P frame following the P frame, and then are further trained through the training process for the P frame following the I frame. can

In another embodiment, the second encoder 1215 and the second decoder 333 are trained through the training process for the P frame following the I frame, and then are further trained through the training process for the P frame following the P frame. can

In another embodiment, the encoder 1215 and the second decoder 333 may be separately trained through a training process for a P frame following an I frame and a training process for a P frame following a P frame. For example, the second encoder 1215 and the second decoder 333 trained through the training process for the P frame after the P frame are applied to the current image after the P frame, and the P frame after the I frame. The second encoder 1215 and the second decoder 333 trained through the training process for

21 is a training process of the first decoder 331, the second decoder 333, the first encoder 1211, the second encoder 1215, and the optical flow prediction neural network 2200 by the training device 2100. It is a drawing for explanation.

The optical flow prediction neural network 2200 is a neural network used to obtain a predicted optical flow, for example, the first prediction neural network 610 shown in FIG. 6 , the second prediction neural network 710 shown in FIG. 7 or It may be the third encoder 1410 and the third decoder 810 shown in FIG. 14 .

The training process described with reference to FIG. 20 may be performed by the training apparatus 2100 . The training apparatus 2100 may be, for example, the image encoding apparatus 1200 or a separate server. The parameters obtained as a result of training are stored in the image encoding apparatus 1200 and the image decoding apparatus 300 .

Referring to FIG. 21 , the training device 2100 includes a first encoder 1211 , a first decoder 331 , a second encoder 1215 , a second decoder 333 , and an optical flow prediction neural network 2200 . is initially set (S2110). Accordingly, the first encoder 1211 , the first decoder 331 , the second encoder 1215 , the second decoder 333 , and the optical flow prediction neural network 2200 may operate according to the initially set parameters. .

The training apparatus 2100 provides the data required for the prediction neural network 2200 of the optical flow to obtain the current prediction optical flow (eg, the previous optical flow, the characteristic data of the previous optical flow, and the characteristic of the previous residual optical flow). at least one of the data) is input to the prediction neural network 2200 of the optical flow (S2115).

The optical flow prediction neural network 2200 processes the input data and outputs the current prediction optical flow to the first encoder 1211 and the training apparatus 2100 (S2120).

The training apparatus 2100 inputs the current training image 2010 and the previous reconstructed training image 2030 to the first encoder 1211 ( S2125 ).

The first encoder 1211 processes the current prediction optical flow, the current training image 2010, and the previous reconstructed training image 2030 to provide feature data (h _i ) of the current residual optical flow to the training apparatus 2100 and It outputs to the first decoder 331 (S2130).

The training apparatus 2100 calculates the second loss information 2070 from the feature data h _i of the current residual optical flow ( S2135 ).

The first decoder 331 processes the feature data h _i of the current residual optical flow and outputs the current residual optical flow d _i to the training apparatus 2100 ( S2140 ).

The training apparatus 2100 generates a current prediction training image (x' _i ) using the current optical flow obtained based on the current prediction optical flow and the current residual optical flow (d _i ), and the current prediction training Current residual image data ri corresponding to the difference between the image x' _i and the current training image 2010 is acquired ( _S2145 ).

The training apparatus 2100 inputs the current residual image data r _i to the second encoder 1215 ( S2150 ), and the second encoder 1215 trains the feature data v _i of the current residual image data. output to the device 2100 and the second decoder 333 (S2155).

The training apparatus 2100 calculates the third loss information 2080 from the feature data v _i of the current residual image data ( S2160 ).

The second decoder 333 processes the feature data v _i of the current residual image data and outputs the current residual image data r′ _i to the training apparatus 2100 ( S2165 ).

The training apparatus 2100 generates a current reconstructed training image 2050 from the current residual image data (r' _i ) and the current prediction training image (x' _i ) ( S2170 ).

The training apparatus 2100 calculates the first loss information 2060 corresponding to the difference between the current training image 2010 and the current reconstruction training image 2050 ( S2180 ).

The training device 2100 calculates final loss information by combining at least one of the first loss information 2060, the second loss information 2070, and the third loss information 2080, and the first decoder 331, the first The second decoder 333, the first encoder 1211, the second encoder 1215, and the optical flow prediction neural network 2200 update the initially set parameters through a back propagation process based on the final loss information ( S2181, S2183, S2185, S2187, S2189).

Thereafter, the training apparatus 2100, the first decoder 331, the second decoder 333, the first encoder 1211, the second encoder 1215, and the optical flow prediction neural network 2200 minimize the final loss information. The parameters are updated while repeating steps S2115 to S2189 until the At this time, during each iteration process, the first decoder 331 , the second decoder 333 , the first encoder 1211 , the second encoder 1215 , and the optical flow prediction neural network 2200 have the parameters updated in the previous process. operates according to

22 is a diagram for explaining a training process of the fourth encoder 1710 , the fourth decoder 1010 , the second encoder 1215 , and the second decoder 333 by the training apparatus 2100 .

If the training process shown in FIG. 21 is a training process for a P frame following a P frame, the training process shown in FIG. 22 may be understood as a training process for a P frame following an I frame.

Referring to FIG. 22 , the training apparatus 2100 initially sets parameters of the fourth encoder 1710 , the fourth decoder 1010 , the second encoder 1215 , and the second decoder 333 ( S2210 ). Accordingly, the fourth encoder 1710 , the fourth decoder 1010 , the second encoder 1215 , and the second decoder 333 may operate according to initially set parameters. According to an embodiment, the second encoder 1215 and the second decoder 333 may initially operate according to parameters set through the training process shown in FIG. 21 .

The training apparatus 2100 inputs the current training image 2010 and the previous reconstructed training image 2030 to the fourth encoder 1710 ( S2215 ).

The fourth encoder 1710 processes the current training image 2010 and the previous reconstructed training image 2030 and outputs feature data of the current optical flow to the training apparatus 2100 and the fourth decoder 1010 (S2220). .

The training apparatus 2100 calculates the second loss information 2070 from the feature data of the current optical flow ( S2225 ).

The fourth decoder 1010 processes the feature data of the current optical flow and outputs the current optical flow to the training apparatus 2100 (S2230).

The training apparatus 2100 generates a current prediction training image (x' _i ) using the current optical flow, and a current corresponding to the difference between the current prediction training image (x' _i ) and the current training image 2010 . The residual image data (r _i ) of is obtained ( S2235 ).

The training apparatus 2100 inputs the current residual image data r _i to the second encoder 1215 ( S2240 ), and the second encoder 1215 trains the feature data v _i of the current residual image data. output to the device 2100 and the second decoder 333 (S2245).

The training apparatus 2100 calculates the third loss information 2080 from the feature data v _i of the current residual image data ( S2250 ).

The second decoder 333 processes the feature data v _i of the current residual image data and outputs the current residual image data r′ _i to the training apparatus 2100 ( S2260 ).

The training apparatus 2100 generates a current reconstruction training image 2050 from the current residual image data (r' _i ) and the current prediction training image (x' _i ) ( S2265 ).

The training apparatus 2100 calculates first loss information 2060 corresponding to a difference between the current training image 2010 and the current reconstruction training image 2050 ( S2270 ).

The training device 2100 calculates final loss information by combining at least one of the first loss information 2060, the second loss information 2070, and the third loss information 2080, and the fourth encoder 1710, the second 4 The decoder 1010, the second encoder 1215, and the second decoder 333 update the initially set parameters through a back propagation process based on the final loss information (S2271, S2273, S2275, S2277).

Thereafter, the training apparatus 2100, the fourth encoder 1710, the fourth decoder 1010, the second encoder 1215, and the second decoder 333 repeat the processes S2215 to S2277 until the final loss information is minimized. while updating the parameters. At this time, during each iteration process, the fourth encoder 1710 , the fourth decoder 1010 , the second encoder 1215 , and the second decoder 333 operate according to the parameters updated in the previous process.

Meanwhile, the above-described embodiments of the present disclosure can be written as a program that can be executed on a computer, and the written program can be stored in a device-readable storage medium.

The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' is a tangible device and only means that it does not contain a signal (eg, electromagnetic wave). It does not distinguish the case where it is stored as For example, the 'non-transitory storage medium' may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided in a computer program product (computer program product). Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store or between two user devices (eg smartphones). It can be distributed directly or online (eg, downloaded or uploaded). In the case of online distribution, at least a portion of the computer program product (eg, a downloadable app) is stored at least in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server. It may be temporarily stored or temporarily created.

In the above, the technical idea of the present disclosure has been described in detail with reference to preferred embodiments, but the technical idea of the present disclosure is not limited to the above embodiments, and those of ordinary skill in the art within the scope of the technical spirit of the present disclosure Various modifications and changes are possible by the person.

Claims (14)

In the optical flow restoration method using AI,
obtaining feature data for a current residual optical flow from a bitstream for a current image;
obtaining the current residual optical flow by applying feature data for the current residual optical flow to a first decoder based on a neural network;
obtaining a current predicted optical flow by using at least one of a previous optical flow, feature data for the previous optical flow, and feature data for a previous residual optical flow; and
and reconstructing a current optical flow using the current residual optical flow and the current predicted optical flow.
According to claim 1,
A method of restoring an optical flow, wherein the current image is reconstructed based on a current prediction image generated from a previous reconstructed image based on the reconstructed current optical flow and current residual image data.
According to claim 1,
The step of obtaining the current prediction optical flow comprises:
and determining the previous optical flow as the current predicted optical flow.
According to claim 1,
The step of obtaining the current prediction optical flow comprises:
Applying at least one of the previous optical flow, the feature data for the previous optical flow, and the feature data for the previous residual optical flow to a first prediction neural network to obtain the current predicted optical flow A method of restoring an optical flow, comprising the steps.
According to claim 1,
The step of obtaining the current prediction optical flow comprises:
At least one of the previous optical flow, the feature data for the previous optical flow, and the feature data for the previous residual optical flow is applied to a second prediction neural network to compare the current predicted optical flow and the previous acquiring a second-order optical flow between optical flows; and
and generating the current predicted optical flow by changing the previous optical flow according to the second-order optical flow.
According to claim 1,
The step of obtaining the current prediction optical flow comprises:
obtaining feature data for a second-order optical flow between the current prediction optical flow and the previous optical flow from the bitstream;
obtaining the second-order optical flow by applying the feature data for the second-order optical flow to a third decoder based on a neural network; and
and generating the current predicted optical flow by changing the previous optical flow according to the second-order optical flow.
According to claim 1,
The feature data for the current residual optical flow is,
A method of restoring an optical flow, which is obtained through entropy decoding and inverse quantization of the bitstream.
According to claim 1,
The first decoder based on the neural network,
first loss information corresponding to a difference between a current training image and a current reconstructed training image corresponding to the current training image; and
The method for restoring an optical flow, which is trained based on second loss information corresponding to entropy of feature data for the current residual optical flow of the current training image.
According to claim 1,
The feature data for the current residual optical flow is,
The method for restoring an optical flow, obtained from the bitstream when the current image corresponds to a P frame following a P (predictive) frame.
10. The method of claim 9,
If the current image corresponds to the P frame after the I (intra) frame,
The method of restoring the optical flow,
obtaining feature data for a current optical flow from a bitstream;
Reconstructing the current optical flow by applying the feature data for the current optical flow to a fourth decoder based on a neural network, further comprising the step of restoring the optical flow.
A computer-readable recording medium in which a program for performing the method of claim 1 is recorded.
In the optical flow restoration apparatus using AI,
an acquisition unit configured to acquire feature data for a current residual optical flow from a bitstream for a current image; and
The current residual optical flow is obtained by applying the feature data for the current residual optical flow to the first decoder based on a neural network, and the previous optical flow, feature data for the previous optical flow, and features for the previous residual optical flow An apparatus for restoring an optical flow, comprising: a predictive decoder configured to obtain a current predicted optical flow using at least one of data, and reconstruct a current optical flow using the current residual optical flow and the current predicted optical flow.
In the optical flow encoding method using AI,
obtaining a current predicted optical flow from at least one of a previous optical flow, feature data for the previous optical flow, and feature data for a previous residual optical flow;
acquiring feature data for a current residual optical flow by applying a current image, a previous reconstructed image, and the current predicted optical flow to a first encoder based on a neural network; and
generating a bitstream corresponding to feature data for the current residual optical flow,
and the current residual optical flow corresponds to a difference between the current optical flow and the current predicted optical flow.
An optical flow encoding apparatus using AI, comprising:
Obtaining a current predicted optical flow from at least one of a previous optical flow, feature data for the previous optical flow, and feature data for a previous residual optical flow, and combining the current image, the previous reconstructed image, and the current predicted optical flow with a neural network a predictive encoder to obtain feature data for a current residual optical flow by applying it to a first encoder based on the present invention; and
A generator for generating a bitstream corresponding to the feature data for the current residual optical flow,
and the current residual optical flow corresponds to a difference between the current optical flow and the current predicted optical flow.

Images