KR20220088277A - Image processing apparatus and method for processing multi-frame thereby - Google Patents

Abstract

a memory and a processor, wherein the processor identifies, in a previous frame, a predicted sample corresponding to a current sample of the current frame, and changes the sample value of the collocated sample of the previous frame according to the sample value of the prediction sample to change the sample value of the current frame. A prediction frame is generated, weights are derived by comparing sample values of the current sample with those of a prediction sample, weighted collocated samples of the prediction frame are weighted, and weighted prediction is performed through a neural network including a convolutional layer. An image processing apparatus according to an embodiment is provided for obtaining a current output frame by processing a frame and a current frame.

Description

Image processing apparatus and multi-frame processing method by the same

The present disclosure relates to the field of processing of images or frames. More specifically, the present disclosure relates to a field of processing multiple images or multiple frames based on a neural network.

Various techniques exist for image processing prior to encoding or display of the image. The image processing technology is a technology that processes all types of information whose input/output is an image, and refers to a technology that processes and transforms an image for secondary applications or to aid human understanding, such as image improvement, emphasis, and compression.

Image processing technologies have been developed based on algorithms, and in recent years, with the development of artificial intelligence technology, a significant portion of image processing is being performed based on artificial intelligence. As a representative example of an artificial intelligence model, a neural network exists.

Neural networks can be trained through training data. The neural network can obtain a desired processing result by processing the image with a weight value set through training. However, the neural network-based image processing so far is not effective for processing time-series related multi-frames.

According to an embodiment, it is a technical task to effectively process frames in consideration of temporal correlation of multi-frames.

An image processing apparatus according to an embodiment includes a memory for storing one or more instructions; and a processor executing the one or more instructions stored in the memory, wherein the processor identifies, from a previous frame, a prediction sample corresponding to a current sample of a current frame, A prediction frame of the current frame is generated by changing a sample value of a collocated sample of the previous frame according to a sample value of the prediction sample, and the sample value of the current sample and the prediction sample A weight is derived by comparing the sample values of the prediction frame, the weight is applied to the collocated sample of the prediction frame, and the weighted prediction frame and the current frame are separated through a neural network including a convolutional layer. processing to obtain the current reconstructed frame.

The processor may identify a sample having a sample value most similar to the sample value of the current sample among the collocated sample of the previous frame and neighboring samples of the collocated sample as the prediction sample. .

The processor performs convolution processing on the current sample and neighboring samples of the current sample with a first predetermined filter kernel to obtain a sample value corresponding to the first filter kernel, and includes a plurality of predetermined second filter kernels. Convolution processing the collocated sample of the previous frame and neighboring samples of the collocated sample to obtain sample values corresponding to the plurality of second filter kernels, and to obtain sample values corresponding to the plurality of second filter kernels identify a sample value most similar to a sample value corresponding to the first filter kernel among the sample values, and a sample corresponding to the identified sample value among the collocated sample of the previous frame and neighboring samples of the collocated sample may be determined as the prediction sample.

In the first filter kernel, a sample corresponding to the current sample may have a preset first value, and other samples may have a value of 0.

In the plurality of second filter kernels, one sample has a preset second value and other samples have a value of 0, and the position of the one sample having the second value is determined by the plurality of second filter kernels. This can be different for different filter kernels.

Signs of the preset first value and the preset second value may be opposite to each other.

The processor is configured to change the sample value of the collocated sample by convolution processing the collocated sample of the previous frame and neighboring samples of the collocated sample with a predetermined third filter kernel, wherein the third filter kernel , a sample corresponding to the prediction sample may have a preset third value, and other samples may have a value of 0.

The weight may be inversely proportional to a difference between a sample value of the current sample and a sample value of the prediction sample.

The processor obtains a previous output frame and a previous feature map output as a result of processing the previous frame by the neural network, and according to a positional relationship between the current sample and a predicted sample in the previous frame, the previous output frame and the previous A prediction output frame and a prediction feature map are generated by changing sample values of collocated samples of a feature map, the weight is applied to the prediction output frame and collocated samples of the prediction feature map, and the weighted prediction output A frame, the weighted prediction feature map, the weighted prediction frame, and the current frame may be input to the neural network.

The previous output frame is a first previous output frame output from the neural network and a second output frame obtained as a result of processing in the neural network a first previous output frame restored through encoding and decoding of the first previous output frame May include previous output frames.

The neural network includes a plurality of sub-neural networks including a first convolutional layer, a second convolutional layer, and a plurality of third convolutional layers, wherein the first convolutional layer of the first sub-neural network includes the weighted prediction output The frame, the weighted prediction frame, and the concatenated result of the current frame are concatenated, and the second convolutional layer of the first sub-neural network convolves the weighted prediction feature map, The plurality of third convolutional layers of the first sub-neural network may sequentially concatenate the result of concatenating the feature map output from the first convolution layer and the feature map output from the second convolution layer. have.

A first convolutional layer of a sub-neural network other than the first sub-neural network is a concatenated result of the weighted prediction frame, the current frame, and intermediate reconstructed frames output from the previous sub-neural network. A second convolutional layer of a sub-neural network other than the first sub-neural network performs convolution processing, and a second convolutional layer of a sub-neural network other than the first sub-neural network convolves an intermediate feature map output from the previous sub-neural network, The plurality of third convolutional layers of the neural network may sequentially concatenate a result of concatenating the feature map output from the first convolutional layer and the feature map output from the second convolutional layer.

The processor may transmit a bitstream generated through encoding of the current output frame to the terminal device.

The processor may reproduce the current output frame through a display.

A multi-frame processing method according to an embodiment includes: identifying a prediction sample corresponding to a current sample of a current frame from a previous frame; generating a prediction frame of the current frame by changing a sample value of a collocated sample of the previous frame according to a sample value of the prediction sample; deriving a weight by comparing the sample value of the current sample with the sample value of the prediction sample; applying the weight to collocated samples of the prediction frame; and processing the weighted prediction frame and the current frame through a neural network including a convolutional layer to obtain a current reconstructed frame.

The image processing apparatus and the multi-frame processing method according to an embodiment may improve processing performance of the current frame by processing the current frame based on the temporal correlation between the current frame and the previous frame.

However, effects achievable by the image processing apparatus and the multi-frame processing method by the image processing apparatus according to an embodiment are not limited to those mentioned above, and other effects that are not mentioned are from the description below. It will be clearly understood by those of ordinary skill in the art.

In order to more fully understand the drawings cited herein, a brief description of each drawing is provided.
1 is a diagram illustrating a configuration of an image processing apparatus according to an exemplary embodiment.
2 is a diagram illustrating a process of processing a current frame according to an embodiment.
3 is a diagram for explaining a convolution operation.
FIG. 4 is a diagram for explaining the motion prediction process shown in FIG. 2 .
5 is an exemplary diagram illustrating a convolution operation applied to a current frame for motion prediction.
6 is an exemplary diagram illustrating a convolution operation applied to a previous frame for motion prediction.
7 is an exemplary diagram illustrating prediction samples corresponding to samples in a current frame.
FIG. 8 is a diagram for explaining the motion compensation process shown in FIG. 2 .
9 is an exemplary diagram illustrating a process of motion compensation for a previous frame using a motion prediction result.
10 is an exemplary diagram illustrating a process of applying a weight to a prediction frame obtained as a result of motion compensation.
11 is a diagram for explaining a method of increasing the number of motion vectors obtained by targeting a downsampled frame.
12 is a diagram illustrating a structure of a first sub-neural network included in a neural network.
13 is a diagram illustrating the structure of the last sub-neural network included in the neural network.
14 is a diagram illustrating an application example of an image processing method according to an embodiment.
15 is a diagram illustrating an application example of an image processing method according to another embodiment.
16 is a diagram illustrating an application example of an image processing method according to another embodiment.
17 is a flowchart of a multi-frame processing method according to an embodiment.

Since the present disclosure can make various changes and can have various embodiments, specific embodiments are illustrated in the drawings, and this will be described through detailed description. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present disclosure.

In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the embodiment are only identification symbols for distinguishing one component from other components.

Also, in this specification, when a component is referred to as “connected” or “connected” to another component, the component may be directly connected to or directly connected to the other component, but the opposite is particularly true. Unless there is a description to be used, it will be understood that it may be connected or connected through another element in the middle.

In addition, in the present specification, components expressed as '~ part (unit)', 'module', etc. are two or more components combined into one component, or two or more components for each more subdivided function. may be differentiated into In addition, each of the components to be described below may additionally perform some or all of the functions of other components in addition to the main functions they are responsible for, and some of the main functions of each component may have different functions. It goes without saying that it may be performed exclusively by the component.

Also, in this specification, a 'frame' may be a still image. For example, a 'frame' may include a still image constituting a moving picture (or video).

Also, in the present specification, a 'sample' refers to data assigned to a sampling position of a frame or a feature map and to be processed. For example, it may be a pixel value in a frame in a spatial domain. A unit including at least one sample may be defined as a block.

Also, in the present specification, a 'current sample' means a specific sample included in a current frame to be processed or a sample to be processed among samples included in the current frame. A 'collocated sample' refers to a sample located at the same location as the current sample among samples included in a frame other than the current frame (eg, previous frame, next frame, output frame, feature map, etc.) .

Also, in the present specification, a 'neural network' is an example of an artificial intelligence model simulating a brain nerve, and is not limited to a neural network model using a specific algorithm.

Also, in the present specification, a 'weighting value' is a value used in a calculation process of each layer constituting the neural network, and may be used, for example, when an input value is applied to a predetermined calculation expression. In general, a weighting value is also referred to as a weight. In the present disclosure, it is used in the calculation process of a neural network to distinguish it from a weight derived in a weight derivation process (refer to 230 of FIG. 1 ) to be described later. The weight to be used is referred to as a weight value. The weight value is a value set as a result of training, and may be updated through separate training data if necessary.

Hereinafter, embodiments according to the technical spirit of the present disclosure will be described in detail in turn.

1 is a diagram illustrating a configuration of an image processing apparatus 100 according to an exemplary embodiment.

The image processing apparatus 100 includes a memory 110 and a processor 130 .

The image processing apparatus 100 may be implemented as a device having an image processing function, such as a server, a TV, a camera, a mobile phone, a tablet PC, and a notebook computer.

Although the memory 110 and the processor 130 are separately illustrated in FIG. 1 , the memory 110 and the processor 130 may be implemented through one hardware module (eg, a chip).

The processor 130 may be implemented as a dedicated processor for neural network-based image processing. Alternatively, the processor 130 may be implemented through a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphic processing unit (GPU). A dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processing unit for using an external memory.

The processor 130 may include a plurality of processors. In this case, it may be implemented as a combination of dedicated processors, or may be implemented through a combination of software and a plurality of general-purpose processors such as an AP, CPU, or GPU.

The processor 130 may include at least one arithmetic logic unit (ALU) for a convolution operation, which will be described later. For a convolution operation, the ALU may include a multiplier that performs a multiplication operation between sample values and an adder that adds the result values of the multiplication.

The memory 110 may store one or more instructions for processing consecutive frames. In an embodiment, the memory 110 may store a neural network used to generate an output frame. When a neural network is implemented in the form of a dedicated hardware chip for artificial intelligence, or as part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU), the neural network is 110) may not be stored.

The processor 130 sequentially processes consecutive frames according to an instruction stored in the memory 110 to obtain consecutive output frames. Here, the continuous frame may refer to frames constituting a moving picture representatively. However, in the present disclosure, consecutive frames do not necessarily constitute one moving picture. In other words, still images photographed separately from each other may be processed by the image processing apparatus 100 according to a predetermined order, an arbitrary order, or an order set by a user.

1 , the image processing apparatus 100 sequentially processes a first frame (X ₁ ) to an nth frame (X _n ) to sequentially process a first output frame (Y ₁ ) to an nth output frame (Y) _n ) can be obtained. In FIG. 1 , t of X _t represents an order in which frames are processed by the image processing apparatus 100 .

The image processing apparatus 100 may obtain a first output frame Y ₁ to an nth output frame Y _n through a pre-trained neural network. Neural networks can be pre-trained to increase frame resolution, remove noise, extend dynamic range, or improve image quality.

For example, when the neural network is trained to increase the resolution of a frame, the image processing apparatus 100 processes the first frame X ₁ to the nth frame X _n based on the neural network to process the first frame X ₁ ) to n-th frame (X _n ) having a resolution greater than that of the first output frame (Y ₁ ) to the n-th output frame (Y _n ) may be obtained. There may be various ways to train a neural network to increase the resolution of a frame. As an example, by comparing the training output frame obtained as a result of processing the training frame by the neural network with the correct answer frame having a pre-increased resolution, loss information is calculated, and the neural network is trained in a direction to minimize the calculated loss information. can As a result of training the neural network, weight values used in layers in the neural network may be updated.

As another example, when the neural network is trained to remove noise from a frame, the image processing apparatus 100 processes the first frame (X ₁ ) to the nth frame (X _n ) based on the neural network to process the first frame (X ₁ ) ) to n-th frame (X _n ) having less noise than noise included in the first output frame (Y ₁ ) to the n-th output frame (Y _n ) may be obtained. There may be various ways to train a neural network to reduce frame noise. For example, by comparing the training output frame obtained as a result of processing the training frame by the neural network with the correct answer frame from which noise has been removed in advance, the loss information is calculated, and the neural network can be trained in a direction to minimize the calculated loss information. have.

The neural network may be trained through supervised learning, unsupervised learning, or reinforcement learning.

According to an embodiment of the present disclosure, when processing a current frame that is a current processing target among consecutive frames, a previous frame is also used. That is, as shown in FIG. 1 , the first frame X ₁ is also input to the image processing apparatus 100 when the second frame X ₂ is to be processed. As will be described later, the first output frame Y ₁ obtained as a result of the processing of the first frame X ₁ and the feature map obtained during the processing of the first frame X ₁ also include the second frame X ₂ and Together, they may be input to the image processing apparatus 100 .

The reason for inputting the previous frame to the image processing apparatus 100 when processing the current frame is to consider temporal correlation between successive frames. By reflecting the information of the previous frame, for example, sample values of the previous frame, in the processing of the current frame, better results can be expected than when only the current frame is processed based on the neural network.

However, when the previous frame is used as it is, there is a possibility that an error may occur in the position of an object included in the current output frame. This is because the positions of the objects commonly included in the previous frame and the current frame captured at different points of time may be different. In other words, when the common object is located at different points between the current frame and the previous frame, the position of the object included in the previous frame is reflected during the processing of the current frame, so that the position of the object included in the current output frame is changed to the current frame. may be different from

Also, due to the movement of the object, an object existing in the previous frame may be occluded in the current frame. Occlusion means a situation in which all or part of an object existing in a previous frame is not included in the current frame. For example, an object included in the previous frame may be hidden by another object in the current frame, or may not be captured by the camera when capturing the current frame. An object of a previous frame that is occluded in the current frame may not be helpful in processing the current frame.

Accordingly, the image processing apparatus 100 according to an embodiment of the present disclosure uses the previous frame together when processing the current frame in order to consider the temporal correlation between the current frame and the previous frame, but does not use the previous frame as it is, but instead uses the previous frame The prediction frame generated from . is used for processing the current frame.

Also, the image processing apparatus 100 may determine to what extent samples of the prediction frame are to be used in the processing of the current frame, and may perform gating processing on the prediction frame, which will be described later.

Although not shown in FIG. 1 , the image processing apparatus 100 may further include a display or be connected to a separate display apparatus. At least one of the continuous output frames generated by the image processing apparatus 100 may be reproduced on a display or a display apparatus. If necessary, at least one of the output frames may be post-processed and then reproduced on a display or display device.

According to an embodiment, the image processing apparatus 100 may encode at least one of the output frames through an image compression method using frequency transformation. An image compression method using frequency transform predicts an output frame to generate prediction data, generates residual data corresponding to a difference between the output frame and prediction data, and transforms the spatial domain component residual data into a frequency domain component It may include a process of transforming, a process of quantizing the residual data transformed into a frequency domain component, and a process of entropy encoding the quantized residual data. This video compression method uses frequency conversion such as MPEG-2, H.264 Advanced Video Coding (AVC), MPEG-4, High Efficiency Video Coding (HEVC), VC-1, VP8, VP9, and AV1 (AOMedia Video 1). It may be implemented through one of the used image compression methods.

The encoded data or bitstream generated by encoding the output frame is transmitted to an external device through a network, magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and floppy disks. The data may be stored in a data storage medium such as a magneto-optical medium such as a optical disk.

Hereinafter, a processing process of the current frame X _t by the image processing apparatus 100 will be described in detail with reference to FIG. 2 .

The image processing apparatus 100 obtains data to be input to the neural network 250 through a motion prediction process 210 , a motion compensation process 220 , a weight derivation process 230 , and a gating process 240 .

First, the motion prediction 210 is a process of determining a motion vector between samples of a current frame (X _t ) and samples of a previous frame (X _t-1 ). The motion vector indicates a relative positional relationship between the same or similar samples existing in the previous frame (X _t-1 ) and the current frame (X _t ). For example, if a specific sample is located at (a,b) coordinates in the previous frame (X _t-1 ) and the corresponding sample is located at (c, d) coordinates in the current frame (X _t ), the sample's The motion vector may be expressed as (ca, db). As will be described later, in an embodiment of the present disclosure, a motion vector may be expressed as a filter kernel for a convolution operation.

The image processing apparatus 100 identifies prediction samples corresponding to samples of the current frame X _t in the previous frame X _t-1 through the motion prediction 210 . Specifically, the image processing apparatus 100 searches which samples of the current frame (X _t ) are similar to which of the samples of the previous frame (X _t-1 ) are similar, and the searched previous frame (X _t-1) ) may be identified as prediction samples of samples in the current frame (X _t ). For example, the image processing apparatus 100 determines that the current sample of the current frame X _t is the same as the current sample among samples of the previous frame X _t-1 (ie, the collocated sample). The sample to the right of the collocated sample can be identified as the predictive sample of the current sample if it is most similar to the sample on the right.

In an embodiment, the image processing apparatus 100 may perform the motion prediction 210 through a convolution operation, which will be described with reference to FIGS. 4 to 7 .

The image processing apparatus 100 uses the relative positional relationship between the samples of the current frame (X _t ) and the corresponding prediction samples in the previous frame (X _t-1 ) as a motion vector in the motion compensation process 220 .

The motion compensation process 220 is a process of obtaining a predicted frame X _{t_pred} , which is a predicted version of the current frame X _t , by changing sample values of samples of the previous frame X _t-1 .

The image processing apparatus 100 may obtain the prediction frame X _{t_pred} by changing sample values of the previous frame X _t-1 according to sample values of the prediction samples. Specifically, sample values of collocated samples of a previous frame (X _t-1 ) at the same position as samples of the current frame (X _t ) may be changed according to sample values of prediction samples. For example, when the prediction sample of the current sample located at the upper left of the current frame (X _t ) is located at the right of the collocated sample of the previous frame (X _t -1 ), the previous frame (X _t-1 ) The sample value of the collocated sample of may change according to the sample value located to the right (ie, the sample value of the prediction sample).

The weight derivation process 230 determines to what extent samples of the prediction frame X _{t_pred} are helpful in processing the current frame X _t or in the prediction frame X _{t_pred} in the processing of the current frame X _t . It can be understood as a process of determining the contribution of samples.

Weights to be applied to samples of the prediction frame X _{t_pred} are derived through the weight derivation process 230 . In processing the current frame (X _t ), a high weight is derived for a sample having a high contribution, and a low weight is derived from a sample having a low contribution to processing the current frame (X t ).

The weight may be based on a difference value between sample values of samples of the current frame (X _t ) and sample values of corresponding prediction samples. The larger the difference value, the smaller the weight, and the smaller the difference value, the larger the weight. A large difference value means that the sample value of the prediction sample does not have a high contribution to the processing of the sample of the current frame X _t , so the weight is calculated to be small.

The gating process 240 is a process of applying a weight to the samples of the prediction frame X _{t_pred} . Sample values of the prediction frame X _{t_pred} are changed according to the contribution of the samples of the prediction frame X _{t_pred} .

In an embodiment, in the gating process 240 , sample values of samples of the prediction frame X _{t_pred} may be multiplied by a weight. The sample value of the sample multiplied by a weight of 1 remains the same, but the sample value of the sample multiplied by a weight less than 1 becomes smaller.

As described with reference to FIG. 1 , the image processing apparatus 100 processes the current frame X _t and the previous output frame Y _{t-1 corresponding to the previous frame X t-1 and} the previous frame X The previous feature map (S _{t-1 )} obtained during the processing of t-1) may be further used.

The previous output frame Y _t-1 and the previous feature map S _t-1 may be output by the neural network 250 in the process of processing the previous frame X _t-1 . The previous output frame Y _t-1 may be output from the last layer of the neural network 250 , and the previous feature map S _t-1 may be output from a previous layer of the last layer of the neural network 250 . Here, the previous layer of the last layer may mean a previous layer directly connected to the last layer or a previous layer in which one or more layers exist between the last layer and the last layer.

Since the previous output frame (Y _t-1 ) and the previous feature map (S _t-1 ) also have the characteristics of the previous frame (X _t-1 ), the above-described motion compensation process 220 and gating process 240 are applied I need this. That is, the prediction output frame (Y _{t_pred} ), which is a predicted version of the current output frame (Y _t ) from the previous output frame (Y _t-1 ) and the previous feature map (S _t-1 ) through the motion compensation process 220, and, A predictive feature map (S _{t_pred} ) that is a predictive version of the current feature map (S _t ) may be obtained.

The motion compensation process 220 for the previous output frame Y _t-1 and the previous feature map S _t-1 is the same as the motion compensation process 220 for the previous frame X _t-1 described above. Specifically, the previous output frame (Y _{t - 1} ) and the previous feature map (Y t-1 ) and the previous feature map ( A prediction output frame (Y _{t_pred} ) and a prediction feature map (S _{t_pred} ) may be generated by changing the sample value of the collocated sample of S _t-1 . For example, if the predicted sample of the current sample of the current frame (X _t ) is located to the right of the collocated sample of the previous frame (X _t-1 ), the previous output frame (Y _t-1 ) and the previous feature map The sample value of the collocated sample of (S _t-1 ) may be changed according to the sample value located to the right of it.

Weights obtained through the weight derivation process 230 are applied to samples of the prediction output frame (Y _{t_pred} ) and the prediction feature map (S _{t_pred} ) in the gating process 240 , so that the weighted prediction output frame (Y' _{t_pred} ) and a weighted prediction feature map (S' _{t_pred} ) may be obtained.

2 shows that both the previous output frame (Y _t-1 ) and the previous feature map (S _t-1 ) are used in processing the current frame (X _t ), but this is only an example, and the previous output frame (Y _t-1 ) and the previous feature map (S _t-1 ) may not be used in the process of processing the current frame (X _t ). That is, when processing the current frame (X _t ), only the previous frame (X _t-1 ) may be considered. As another example, when processing the current frame (X _t ), only one of the previous output frame (Y _t-1 ) and the previous feature map (S _t-1 ) may be used.

The weighted prediction frame (X' _{t_pred} ), the weighted prediction output frame (Y' _{t_pred} ), the weighted prediction feature map (S' _{t_pred} ) and the current frame (X _t ) derived through the gating process 240 are 250) is entered. As a result of the processing of the neural network 250 , the current output frame Y _t corresponding to the current frame X _t is obtained.

The neural network 250 according to an embodiment may include a convolutional layer. In the convolution layer, convolution processing is performed on input data using a filter kernel. Convolution processing for the convolution layer will be described with reference to FIG. 3 .

The neural network 250 may include one or more sub-neural networks 260-1 ... 260-n. 2 shows that several sub-neural networks 260-1 ... 260-n are included in the neural network 250, but this is only an example and only one sub-neural network 260-1 is included in the neural network 250. may be included. The fact that one sub-neural network 260-1 is included in the neural network 250 may mean that the neural network 250 includes a fusion layer 262 and a plurality of convolutional layers.

The fusion layer includes a current frame (X _t ), data output from the gating process 240 , that is, a weighted prediction frame (X' _{t_pred} ), a weighted prediction output frame (Y' _{t_pred} ) and weighted Fuse the prediction feature map (S' _{t_pred} ). Through the fusion process, different types of data can be integrated.

The result of integrating the weighted prediction frame (X' _{t_pred} ), the weighted prediction output frame (Y' _{t_pred} ), the weighted prediction feature map (S' _{t_pred} ) and the current frame (X _t ) by the fusion layer 262 is Convolution is performed by subsequent convolutional layers 264 .

As a result of processing by the first sub-neural network 260-1, an intermediate output frame Y _{t_int} and an intermediate feature map S _{t_int} are obtained. The intermediate output frame (Y _{t_int} ) is output by the last layer included in the first sub-neural network 260-1, and the intermediate feature map (S _{t_int} ) is the last layer included in the first sub-neural network 260-1. output by the previous layer.

The current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), and the intermediate output frame (Y _{t_int} ) and the intermediate feature map (S _{t_int} ) output from the first sub-neural network 260-1 are the second sub-neural network It is entered as (260-2). Like the first sub-neural network 260-1, in the fusion layer 262 of the second sub-neural network 260-2, the current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), and the intermediate output frame (Y) _{t_int} ) and the intermediate feature map (S _{t_int} ) are integrated and then convolved. As a result of the processing of the second sub-neural network 260-2, an intermediate output frame (Y _{t_int} ) and an intermediate feature map (S _{t_int} ) are output, and the output intermediate output frame (Y _{t_int} ) and the intermediate feature map (S _{t_int} ) are three It is input to the second sub neural network 260 - 3 . Similar to the second sub-neural network 260 - 2 , the current frame X _t and the weighted prediction frame X' _{t_pred} may be further input to the third sub neural network 260 - 3 . As a result of processing by the last sub-neural network 260-n, the current output frame Y _t corresponding to the current frame X _t is obtained.

The current frame (X _t ), the current output frame (Y _t ) output from the last sub-neural network 260-n, and the current feature map (S _t ) may be used in the processing of the next frame (X _t+1 ). .

On the other hand, when the current frame (X _t ) is the first frame of a continuous frame, the previous frame (X _t-1 ), the previous output frame (Y _t-1 ), and the previous feature map (S _t-1 ) are predetermined It can be set to have a sample value (eg, 0).

Hereinafter, a convolution operation will be described before the motion prediction process 210 and the motion compensation process 220 are described in detail.

3 is a diagram for explaining a convolution operation.

The feature map 350 is generated through multiplication and addition operations between weight values of the filter kernel 330 used in the convolution layer and sample values in the frame 310 corresponding thereto. The filter kernel 330 has a predetermined size (3x3 in FIG. 3).

The number of feature maps 350 varies according to the number of filter kernels 330 . The number of filter kernels 330 and the number of feature maps 350 may be the same. That is, when one filter kernel 330 is used in the convolution layer, one feature map 350 may be generated, and if two filter kernels 330 are used, two feature maps 350 may be generated.

In FIG. 3 , I1 to I49 indicated in the frame 310 indicate samples of the frame 310 , and F1 to F9 indicated in the filter kernel 330 indicate weight values of the filter kernel 330 . Also, M1 to M9 displayed in the feature map 350 represent samples of the feature map 350 .

3 illustrates that the frame 310 includes 49 samples, but this is only an example, and when the frame 310 has a resolution of 4K, for example, it may include 3840 X 2160 samples. can

In the convolution operation process, each of the sample values of I1, I2, I3, I8, I9, I10, I15, I16, and I17 of the frame 310 and F1, F2, F3, F4, F5, F6 of the filter kernel 330 is performed. , F7, F8, and F9 may each be multiplied, and a value obtained by combining (eg, addition operation) result values of the multiplication operation may be assigned as the value of M1 of the feature map 350 . If the stride of the convolution operation is 2, each of the sample values of I3, I4, I5, I10, I11, I12, I17, I18, I19 of the frame 310 and F1, F2, F3 of the filter kernel 330 , F4, F5, F6, F7, F8, and F9 may each be multiplied, and a value obtained by combining result values of the multiplication operation may be assigned as the value of M2 of the feature map 350 .

A convolution operation is performed between the sample values in the frame 310 and the weight values of the filter kernel 330 while the filter kernel 330 moves along the stride until the last sample of the frame 310 is reached. A feature map 350 having a size may be obtained.

FIG. 4 is a diagram for explaining the motion prediction process 210 shown in FIG. 2 .

In an embodiment, the image processing apparatus 100 may perform motion prediction based on the convolution operations 410 and 420 on the current frame X _t and the previous frame X _t-1 .

The image processing apparatus 100 obtains a first feature map 417 by performing a convolution operation 410 on the current frame X _t with a first predetermined filter kernel 415 , and a plurality of predetermined first filter kernels 415 . A plurality of second feature maps 427 may be obtained by performing a convolution operation 420 on the previous frame X _t−1 using the two filter kernels 425 .

A convolution operation 410 based on the first filter kernel 415 may be sequentially performed on samples of the current frame X _t , and a convolution operation based on a plurality of second filter kernels 425 ( 420 may be sequentially performed on samples of the previous frame (X _t−1 ).

The convolution operation 410 based on the first filter kernel 415 is performed on the current sample of the current frame X _t and neighboring samples of the current sample, so that the collocated sample of the first feature map 415 is Sample values may be obtained. In addition, the convolution operation 420 based on the plurality of second filter kernels 425 is performed on the collocated sample of the previous frame (X _t-1 ) and neighboring samples of the collocated sample, so that a plurality of A sample value of the collocated sample of the second feature maps 427 may be obtained.

The first filter kernel 415 and the plurality of second filter kernels 425 may have a predetermined size. For example, the first filter kernel 415 and the plurality of second filter kernels 425 may have a size of 3x3 as shown in FIG. 4 , but is not limited thereto. Depending on the implementation, the first filter kernel 415 and the plurality of second filter kernels 425 may have a size of 4x4 or 5x5.

As a result of the convolution operation 410 for the current frame X _t based on the first filter kernel 415 , a first feature map 417 having the same size as the current frame X _t may be obtained. The current frame X _t may be padded to obtain the first feature map 417 having the same size as the current frame X _t through the convolution operation 410 . Padding means allocating a sample having a predetermined sample value (eg, 0) outside at least one of a left boundary, an upper boundary, a right boundary, and a lower boundary of the current frame (X _t ). The number of samples of the current frame (X _t ) is increased through padding.

In the first filter kernel 415 , a sample corresponding to the current sample may have a preset first value, and the remaining samples may have a value of zero. Although FIG. 4 shows that the first filter kernel 415 has a value of 1 and a value of 0, 1 is only an example of a preset first value.

The first filter kernel 415 is applied to the current sample of the current frame (X _t ) and neighboring samples of the current samples. It means the sample applied to the multiplication operation. 4 shows that the central sample of the first filter kernel 415 has a weight of 1, which means that the current frame X _t is padded to the left of the left boundary and toward the top of the upper boundary. case is taken into account. For example, when the convolution operation 410 is performed on the upper-left sample of the current frame (X _t ), the upper-left sample and the first filter kernel must be padded in the left and upper directions of the current frame (X _t ). A multiplication operation between the central samples of (415) may be performed. Accordingly, if the current frame X _t is not padded in the left direction and the upper direction, the upper left sample of the first filter kernel 415 has a value of 1.

Among the samples of the first filter kernel 415 , a sample corresponding to the current sample has a preset first value, so that the samples of the first feature map 417 are added to sample values of samples of the current frame X _t . It is calculated by multiplying the value by 1. Accordingly, if the first value is 1, the sample values of the first feature map 417 are equal to the sample values of the current frame X _t .

According to an embodiment, the convolution operation 410 for the current frame (X _t ) may be omitted in the motion prediction process. The reason is that when the sample of the first filter kernel 415 corresponding to the current sample has a value of 1 and other samples have a value of 0, the first feature obtained as a result of the convolution operation 410 is This is because the map 417 is eventually identical to the current frame X _t . Accordingly, the prediction samples 430 can be identified through comparison of the second feature maps 427 obtained as a result of the convolution operation 420 for the previous frame X _t-1 and the current frame X _t . have.

The plurality of second filter kernels 425 used in the convolution operation 420 for the previous frame X _t-1 have a value of 0 and a preset second value. The preset second value may be the same as the preset first value. For example, both the first value and the second value may be 1. When the first value and the second value are the same, the first filter kernel 415 may correspond to any one of the plurality of second filter kernels 425 .

According to an embodiment, the preset second value may have a different sign from the preset first value. For example, if the first value is 1, the second value may be -1.

Any one of the samples of the plurality of second filter kernels 425 may have a preset second value, and the remaining samples may have a value of 0. The position of the sample having the second value may be different for each of the plurality of second filter kernels 425 . As shown in FIG. 4 , in any one of the second filter kernels 425 , the upper left sample has the second value, and in the other second filter kernel 425 , the right sample of the upper left sample has the second value. can have

The number of second filter kernels 425 may vary according to the size of the second filter kernels 425 . If the size of the second filter kernels 425 is 3x3, the number of the second filter kernels 425 may be nine. This is because the position of the sample having the second value is different for each of the second filter kernels 425 .

Second feature maps 427 are obtained through a convolution operation 420 on the previous frame X _t-1 using the second filter kernels 425 . The number of second feature maps 427 is equal to the number of second filter kernels 425 .

The second filter kernels 425 may be used to extract a sample value of any one of the collocated sample of the previous frame (X _t−1 ) and neighboring samples of the collocated sample. For example, the second filter kernel in which the upper-left sample has a second value may be used to extract a sample value of a sample located in the upper-left corner of the collocated sample of the previous frame (X _t-1 ), The second filter kernel in which the side sample has a second value may be used to extract a sample value of a sample located at the upper right of the collocated sample of the previous frame (X _t-1 ).

The image processing apparatus 100 compares sample values of the first feature map 417 with sample values of the second feature maps 427 to identify predicted samples 430 of samples in the current frame X _t . have. The image processing apparatus 100 identifies a sample most similar to a sample value of a predetermined location of the first feature map 417 among sample values of a predetermined location of the second feature maps 427 and a previous frame corresponding to the checked sample A sample within (X _t-1 ) may be identified as a prediction sample 430 of the current sample at a predetermined position.

For example, if the current sample is a center sample within the current frame X _t , a sample value most similar to a sample value of the center sample of the first feature map 417 among the sample values of the center samples of the second feature maps 427 . This is confirmed. In addition, a sample in the previous frame (X _t-1 ) corresponding to the checked sample value may be identified as the prediction sample 430 of the current sample. If the sample value of the center sample of the second feature map 427 generated based on the second filter kernel 425 in which the right-right sample has the second value is the sample value of the center sample of the first feature map 417 . , the sample located at the upper right of the center sample of the previous frame (X _t−1 ) may be determined as the prediction sample 430 of the current sample.

Hereinafter, a motion prediction process will be described with reference to FIGS. 5 to 7 as an example.

5 is an exemplary diagram illustrating a convolution operation applied to a current frame 510 for motion prediction.

In the first filter kernel 415 , the central sample corresponding to the current sample has a value of 1, and the remaining samples have a value of 0.

The current frame 510 may have samples of a1, b1, c1, d1, e1, f1, g1, h1, and i1. 5 shows that the size of the current frame 510 is 3x3, this is only for convenience of description, and the size of the current frame 510 may vary.

In order to generate the first feature map 417 having the same size as the current frame 510 , the current frame 510 may be padded in the left direction, the upper direction, the right direction, and the lower direction. Samples p0 to p15 having predetermined sample values in a left direction, an upper direction, a right direction, and a lower direction of the current frame 510 may be added to the current frame 510 through padding.

In order to sequentially perform a convolution operation on samples of the current frame 510 , the stride of the convolution operation may be set to 1.

First, a convolution operation of the weight values of the p0 sample, p1 sample, p2 sample, p5 sample, a1 sample, b1 sample, p7 sample, d1 sample, and e1 sample of the current frame 510 and the first filter kernel 415 is performed. A sample value of the first sample (ie, the upper left sample) of the first feature map 417 is derived through this. Since the center sample of the first filter kernel 415 has a value of 1 and the remaining samples have a value of 0, the sample value of the upper-left sample of the first feature map 417 is derived as a1.

Next, a convolution operation of the weight values of the p1 sample, p2 sample, p3 sample, a1 sample, b1 sample, c1 sample, d1 sample, e1 sample, and f1 sample of the current frame 510 and the first filter kernel 415 is performed. A sample value of the second sample of the first feature map 417 (ie, a sample located to the right of the upper-left sample) is derived through . A sample value of the second sample of the first feature map 417 is derived as b1 through a convolution operation.

A convolution operation based on the samples of the current frame 510 and the first filter kernel 415 is performed until the first filter kernel 415 reaches the last sample of the current frame 510 , that is, the i1 sample. When the first filter kernel 415 arrives at the i1 sample, the e1 sample, the f1 sample, the p8 sample, the h1 sample, the i1 sample, the p10 sample, the p13 sample, the p14 sample and the p15 samples and the first sample of the current frame 510 . A sample value of the last sample of the first feature map 417 is derived as i1 through a convolution operation of the weight values of the filter kernel 415 .

Referring to FIG. 5 , when the central sample of the first filter kernel 415 has a value of 1, it can be seen that the sample values of the current frame 510 and the sample values of the first feature map 417 are the same. have. That is, the first filter kernel 415 in which a sample corresponding to the current sample has a value of 1 is used to extract sample values of the current frame 510 .

6 is an exemplary diagram illustrating a convolution operation applied to a previous frame for motion prediction.

The second filter kernels 425 may include one sample having a value of 1 and other samples having a value of 0. As described above, a position of a sample having a weight of 1 may be different for each second filter kernel 425 . The second filter kernels 425 may include one sample having a value of -1 and other samples having a value of 0.

The previous frame 530 may include samples of a2, b2, c2, d2, e2, f2, g2, h2, and i2. 6 illustrates that the size of the previous frame 530 is 3x3, this is only for convenience of description, and the size of the previous frame 530 may vary.

In order to generate the second feature maps 427 having the same size as the previous frame 530 , the previous frame 530 may be padded in the left direction, the upper direction, the right direction, and the lower direction. Samples having predetermined sample values in a left direction, an upper direction, a right direction, and a lower direction of the previous frame 530 may be added to the previous frame 530 through padding.

Second feature maps 427 corresponding to the second filter kernels 425 may be obtained through a convolution operation based on the previous frame 530 and each of the second filter kernels 425 .

Hereinafter, each of the second filter kernels 425 is referred to as a second filter kernel (A) 425-1 and a second filter according to a location of a sample having a value of 1 to distinguish the second filter kernels 425. Referring to as a kernel (B) 425-2 to a second filter kernel (I) 425-9, each of the second feature maps 427 is used to distinguish between the second feature maps 427 as a second feature. Reference is made to the map (A) 427-1, the second feature map (B) 427-2 to the second feature map (I) 427-9.

In order to sequentially perform a convolution operation on samples of the previous frame 530 , the stride of the convolution operation may be set to 1.

First, a second feature map (A) 427-1 may be obtained through a convolution operation based on the second filter kernel (A) 425-1 having the upper-left sample value of 1 and the previous frame 530. have. As described above, the second filter kernel (A) 425 - 1 may be convolved with the previous frame 530 while moving along the stride of 1. The second filter kernel (A) 425 - 1 of the samples (a2 sample, b2 sample, c2 sample, d2 sample, e2 sample, f2 sample, g2 sample, h2 sample, i2 sample) of the previous frame 530 ) This is to extract the sample value located in the upper left corner. Accordingly, the second feature map (A) 427 - 1 is the samples (a2 sample, b2 sample, c2 sample, d2 sample, e2 sample, f2 sample, g2 sample, h2 sample, i2 sample) of the previous frame 530 . sample) has a value obtained by multiplying the sample value of the samples located in the upper left corner by 1. For example, if the current sample is a1, the collocated sample of the previous frame 530 becomes a2, and the sample value of the collocated sample of the second feature map (A) 427-1 is to the left of the a2 sample. It is derived as a sample value of p0 located at the top.

Next, a second feature map (B) ( 427-2) can be obtained. The second filter kernel (B) 425 - 2 of the samples of the previous frame 530 (a2 sample, b2 sample, c2 sample, d2 sample, e2 sample, f2 sample, g2 sample, h2 sample, i2 sample) This is to extract the sample value located on the top. Accordingly, the second feature map (B) 427 - 2 is the samples (a2 sample, b2 sample, c2 sample, d2 sample, e2 sample, f2 sample, g2 sample, h2 sample, i2 sample) of the previous frame 530 . It has a value obtained by multiplying the sample value of the samples located above the sample) by 1. For example, if the current sample is a1, the collocated sample of the previous frame 530 becomes a2, and the sample value of the collocated sample of the second feature map (B) 427-2 is the upper part of the a2 sample. It is derived as a sample value of p1 located in .

In this way, the second feature map (A) 427 through the convolution operation of each of the second filter kernel (A) 425-1 to the second filter kernel (I) 425-9 and the previous frame 530 . -1) to the second feature map (I) 427-9 may be obtained.

7 is an exemplary diagram illustrating prediction samples 430 corresponding to samples in a current frame.

The image processing apparatus 100 identifies which sample in the first feature map 417 is most similar to which sample among the samples in the second feature maps 427 . In this case, samples at the same location among the samples of the first feature map 417 and the samples of the second feature maps 427 are compared. Specifically, the image processing apparatus 100 calculates an absolute value of a difference between a sample value of a sample at a specific location in the first feature map 417 and sample values of samples at a specific location in the second feature maps 427 . and a sample value having the smallest calculated absolute value can be identified. The image processing apparatus 100 may determine a sample in the previous frame 530 corresponding to a sample value having the smallest absolute value of the difference as the prediction sample.

As described above, if the sign of the first value of any one sample of the first filter kernel 415 and the sign of the second value of any one sample of the second filter kernel 425 are the same, the first characteristic The difference between the sample value of the sample in the map 417 and the sample values of the samples in the second feature maps 427 may be calculated through a difference operation, and conversely, the sign of the first value and the sign of the second value are If different, a difference between the sample values of the samples in the first feature map 417 and the sample values of the samples in the second feature maps 427 may be calculated through a sum operation.

Referring to the upper-left sample of the first feature map 417 illustrated in FIG. 5 , the image processing apparatus 100 generates a sample value of the upper-left sample a1 of the first feature map 417 and the second feature map 427 . It is possible to calculate the absolute value of the difference between the sample values of the upper-left sample of . For example, |a1-p0| is calculated between the sample value of the upper-left sample a1 of the first feature map 417 and the sample value of the upper-left sample p0 of the second feature map (A) 427-1, |a1-p1| may be calculated between the sample value of the upper-left sample a1 of the first feature map 417 and the sample value of the upper-left sample p1 of the second feature map (B) 427-2. If the absolute value of the difference between the sample value of the upper-left sample e2 of the second feature map (I) 427-9 and the sample value of the upper-left sample a1 of the first feature map 417 is the smallest, the second feature The sample e2 in the previous frame 530 corresponding to the sample e2 of the upper left of the map (I) 427 - 9 may be determined as the prediction sample of the upper left sample a1 of the current frame 510 .

As such, by comparing sample values of the first feature map 417 with sample values of the second feature maps 427 , prediction samples 430 corresponding to each sample of the current frame 510 may be identified. .

7 , it can be seen that prediction samples corresponding to samples of the current frame 510 are determined to be b2, e2, f2, e2, f2, i2, h2, e2, and i2.

Although the motion prediction process has been described with reference to the process of determining the prediction sample in FIGS. 4 to 7 , the motion prediction process may be understood as a process of finding a motion vector. Each of the aforementioned second filter kernels 425 may be interpreted as a positional relationship between the collocated sample of the previous frame 530 and its neighboring samples and the current sample, that is, a motion vector candidate. In other words, the motion prediction process determines a motion vector candidate (one second filter kernel) indicating a sample most similar to the current sample among several motion vector candidates (a plurality of second filter kernels) as the motion vector of the current sample. It can be a process

FIG. 8 is a diagram for explaining the motion compensation process 220 shown in FIG. 2 .

The motion compensation process is a process of changing sample values of samples of a previous frame (X _t-1 ) at the same position as samples of a current frame (X _t ) according to sample values of prediction samples. A prediction frame (X _{t_pred} ) may be obtained through a motion compensation process.

The image processing apparatus 100 may perform a motion compensation process through a convolution operation similar to the motion prediction process.

The image processing apparatus 100 selects a third filter kernel to be used for motion compensation of each sample of the previous frame (X _t-1 ) from among a plurality of predetermined third filter kernels 815, and applies the selected third filter kernel to the selected third filter kernel. A convolution operation based on this can be applied to each sample of the previous frame (X _t-1 ). In this case, a third filter kernel corresponding to each sample of the previous frame (X _t-1 ) may be selected.

The plurality of third filter kernels 815 may include a sample having a third predetermined value and samples having a zero value, and the location of the sample having the third value may be different for each third filter kernel 815 . . The third value may be, for example, one. According to an embodiment, the plurality of second filter kernels 425 used in the motion prediction process may also be used in the motion compensation process.

9 is an exemplary diagram illustrating a process of motion compensation for a previous frame using a motion prediction result.

Reference is made to the third filter kernel (A) 815-1 to the third filter kernel (I) 815-9 according to the location of the sample having the third value to distinguish each of the third filter kernels 815 . do.

The image processing apparatus 100 generates a third filter kernel 815 having a third value at a position corresponding to the prediction sample with respect to each of the samples of the previous frame 530 located at the same position as the samples of the current frame 510 . ) can be selected.

First, the a2 sample located at the upper left of the previous frame 530 will be described as a reference. If the prediction sample of the a1 sample located at the upper left of the current frame 510 is determined to be the b2 sample, the image processing apparatus 100 performs the sample having a value of 1 and the remaining samples having a value of 0 while located to the right of the central sample. The included third filter kernel (F) 815 - 6 may be selected for the a2 sample. In this case, the image processing apparatus 100 includes the p0 sample, the p1 sample, the p2 sample, the p5 sample, the a2 sample, the b2 sample, the p7 sample, the d2 sample, and the e2 sample of the previous frame 530 and the third filter kernel (F). The upper-left sample b2 of the prediction frame 900 may be derived through multiplication and sum operations based on 0, 0, 0, 0, 0, 1, 0, 0, and 0 in (815-6). That is, it can be seen that the a2 sample is replaced with the b2 sample in the prediction frame 900 through a convolution operation on the a2 sample of the previous frame 530 and its neighboring samples.

Next, if the prediction sample of the b1 sample located above the center sample of the current frame 510 is determined to be the e2 sample, the sample having a value of 1 and the remaining samples having a value of 0 while located below the center sample are combined. The containing third filter kernel (H) 815 - 8 may be selected for the b2 sample. The image processing apparatus 100 includes the p1 sample, the p2 sample, the p3 sample, the a2 sample, the b2 sample, the c2 sample, the d2 sample, the e2 sample, the f2 sample, and the third filter kernel (H) 815 of the previous frame 530 . 8), the sample e2 positioned above the center sample of the prediction frame 900 may be derived through the multiplication and sum operations based on 0, 0, 0, 0, 0, 0, 0, 1, and 0. That is, it can be seen that the b2 sample of the previous frame 530 is replaced with the e2 sample in the prediction frame 900 through a convolution operation on the b2 sample of the previous frame 530 and its neighboring samples.

As a convolution operation is performed based on the third filter kernel 815 corresponding to each sample from the first sample to the last sample of the previous frame 530, the prediction frame 900, which is the prediction version of the current frame 510, is can be created

10 is an exemplary diagram illustrating a process of applying a weight 950 to a prediction frame 900 obtained as a result of motion compensation.

The image processing apparatus 100 calculates a weight 950 based on a difference value between a current sample in the current frame 510 and a predicted sample in the previous frame 530 (or a collocated sample in the prediction frame 900 ). can do. The image processing apparatus 100 may calculate a weight 950 for each sample of the current frame 510 .

As described above, the weight 950 indicates to what extent the samples of the prediction frame 900 help the processing of the current frame 510 .

The weight 950 may be derived based on Equation 1 below.

[Equation 1]

는 미리 결정된 상수로서 예를 들어 16일 수 있다. 수학식 1을 참조하면, 현재 샘플의 샘플 값과 예측 샘플의 샘플 값이 동일하다면, 가중치(950)는 1로 산출되고, 현재 샘플의 샘플 값과 예측 샘플의 샘플 값의 차이 값이 커질수록 가중치(950)는 작게 산출되는 것을 알 수 있다.In Equation 1,

may be, for example, 16 as a predetermined constant. Referring to Equation 1, if the sample value of the current sample and the sample value of the prediction sample are the same, the weight 950 is calculated as 1, and as the difference between the sample value of the current sample and the sample value of the prediction sample increases, the weight It can be seen that 950 is calculated to be small.

The image processing apparatus 100 may obtain the weighted prediction frame 1000 by multiplying each sample of the prediction frame 900 by a weight 950 corresponding thereto.

As described above, the image processing apparatus 100 may obtain a weighted prediction output frame and a weighted prediction feature map by applying a weight 950 corresponding thereto to each sample of the prediction output frame and the prediction feature map. .

As described above, in an embodiment, the motion prediction process and the motion compensation process may be performed based on a convolution operation. As shown in Fig. 4, the motion prediction process can be performed once (when the convolution operation 410 for the current frame is omitted) or through two convolution operations, and as shown in Fig. 8, Since the motion compensation process can be performed through one convolution operation, the complexity of the operation can be significantly reduced.

Meanwhile, the above-described motion prediction process may be applied to the downsampled current frame and the downsampled previous frame. This is to reduce the load and complexity of the motion prediction process. Here, the downsampling means a process of reducing the number of samples in a frame. Downsampling of the frame may be performed in various ways. For example, the current frame and the previous frame may be pooled to reduce the number of samples in the current frame and the previous frame. The pooling process may include a max pooling process or an average pooling process. Since the pooling process can be clearly understood from the pooling layer used in the field of artificial neural networks, a detailed description thereof will be omitted. Depending on the implementation, the downsampling of the current frame and the previous frame may be performed through various known downsampling algorithms.

When a motion prediction process is performed with respect to the downsampled current frame and the downsampled previous frame, motion vectors equal to the number of samples included in the downsampled current frame are derived. Since the number of motion vectors required in the motion compensation process is greater than the number of motion vectors acquired through the downsampled frame-based motion prediction process, the number of motion vectors acquired in the motion prediction process should be increased.

A method of increasing the number of motion vectors obtained in the motion prediction process will be described with reference to FIG. 11 .

11 is a diagram for explaining a method of increasing the number of motion vectors obtained by targeting a downsampled frame.

Referring to FIG. 11 , the size of the downsampled frame 1110 is 2x2, and the size of the downsampled frame 1130 is 4x4. The size of the downsampled frame 1110 may be variously changed according to the downsampling ratio.

When the above-described motion prediction process is applied to the downsampled frame 1110 , four motion vectors (ie, filter kernels) corresponding to the four samples included in the frame 1110 are derived. Since the size of the frame 1130 before downsampling is 4x4, 16 motion vectors are required in the motion compensation process.

As an example, the image processing apparatus 100 may group samples of the downsampled frame 1130 according to the number of samples in the downsampled frame 1110 . In addition, the image processing apparatus 100 may allocate each of the motion vectors derived in the motion prediction process to sample groups of the frame 1130 before downsampling. In this case, sample groups of the frame 1130 before downsampling and positions of samples in the downsampled frame 1110 may be considered.

In detail, the motion vector mv1 derived for the upper left sample 1112 among the samples 1112 , 1114 , 1116 , and 1118 in the downsampled frame 1110 is a sample of the frame 1130 before being downsampled. Among the groups 1132 , 1134 , 1136 , and 1138 , the sample group 1132 may be allocated to the upper left. Accordingly, motion compensation may be performed on the samples included in the sample group 1132 positioned at the upper left of the frame 1130 based on the motion vector mv1. Here, it is noted that motion compensation is performed on the previous frame before downsampling.

The motion vector mv2 derived for the upper right sample 1114 among the samples 1112 , 1114 , 1116 , and 1118 in the downsampled frame 1110 is the sample groups 1132 of the frame 1130 before being downsampled. , 1134 , 1136 , and 1138 , may be allocated to the sample group 1134 positioned at the upper right. Accordingly, motion compensation may be performed on samples included in the sample group 1134 positioned at the upper right of the frame 1130 based on the motion vector mv2.

When the number of samples included in the sample group is large, applying the same motion vector to all samples included in the sample group may reduce motion compensation accuracy.

As another example, for samples included in the sample group that are adjacent to the boundary with the adjacent sample group, the image processing apparatus 100 may generate a motion vector allocated to the corresponding sample group and a motion vector allocated to the adjacent sample group. A combined motion vector may be applied.

As another example, the image processing apparatus 100 may obtain motion vectors for motion compensation of the downsampled frame 1130 by interpolating motion vectors obtained with respect to the downsampled frame 1110 . As an example of interpolation, bilinear interpolation, bicubic interpolation, or near-neighbor interpolation may be applied.

When motion prediction is performed on the downsampled frame 1110 , the number of weights derived in the weight derivation process is also smaller than the number of weights required in the gating process. Accordingly, the image processing apparatus 100 increases the number of weights obtained through the weight derivation process. Here, it is noted that the gating process is applied to the prediction frame generated through the motion compensation process from the previous frame before downsampling.

As an example, the image processing apparatus 100 may group samples of the downsampled frame 1130 according to the number of samples in the downsampled frame 1110 . In addition, the image processing apparatus 100 may allocate each of the weights derived in the weight derivation process to sample groups of the frame 1130 before downsampling. In this case, sample groups of the frame 1130 before downsampling and positions of samples in the downsampled frame 1110 may be considered.

In detail, the first weight derived for the upper left sample 1112 among the samples 1112 , 1114 , 1116 , and 1118 in the downsampled frame 1110 is the sample groups of the frame 1130 before being downsampled. Among (1132, 1134, 1136, 1138), the sample group 1132 located at the upper left may be allocated. Accordingly, a gating process may be performed on the samples included in the sample group 1132 positioned at the upper left of the frame 1130 based on the first weight. In addition, the second weight derived for the upper right sample 1114 among the samples 1112 , 1114 , 1116 , and 1118 in the downsampled frame 1110 is the sample groups 1132 of the frame 1130 before being downsampled. , 1134 , 1136 , and 1138 , may be allocated to the sample group 1134 positioned at the upper right. Accordingly, a gating process may be performed on the samples included in the sample group 1134 positioned at the upper right of the frame 1130 based on the second weight.

As another example, the image processing apparatus 100 may set a weight obtained by combining a weight assigned to a corresponding sample group and a weight assigned to an adjacent sample group with respect to samples included in the sample group that are adjacent to the boundary with the adjacent sample group. may be applied.

As another example, the image processing apparatus 100 may obtain weights for the gating process of the downsampled frame 1130 by interpolating the weights obtained with respect to the downsampled frame 1110 . As an example of interpolation, bilinear interpolation, bicubic interpolation, or near-neighbor interpolation may be applied.

Although it has been described above that the motion prediction process and the motion compensation process are performed based on a convolution operation, this is only an example. The motion prediction process and the motion compensation process may be performed by an algorithm used in inter prediction of a known video codec.

For example, the motion prediction process may be performed based on a block matching algorithm or an optical flow algorithm. The block matching algorithm and optical flow are algorithms that search a previous frame for a sample or block most similar to a sample or block in the current frame. A motion vector between a sample or block in the current frame and a similar sample or similar block in the previous frame is obtained through a block matching algorithm and optical flow, and a prediction frame can be obtained by motion compensation for the previous frame based on the obtained motion vector. . A block matching algorithm and an optical flow algorithm are techniques known to those skilled in the art, and detailed descriptions thereof will be omitted herein.

Hereinafter, a neural network used for processing frames will be described in detail with reference to FIGS. 12 and 13 .

As described above with reference to FIG. 2 , the current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), the weighted prediction output frame (Y' _{t_pred} ), and the weighted prediction feature map (S' _{t_pred} ) are the neural networks By processing by 250 , the current output frame Y _t corresponding to the current frame X _t may be obtained.

The neural network 250 may include at least one sub-neural network, and each sub-neural network may include a fusion layer and a plurality of convolutional layers.

The structure of the first sub-neural network 1200 among at least one sub-neural network is illustrated in FIG. 12 .

Referring to FIG. 12 , the first sub-neural network 1200 includes a fusion layer 1210 including a first convolutional layer 1214 and a second convolutional layer 1216 , and a plurality of third convolutional layers 1230 . ) can be composed of In the convolution layer, a convolution operation is performed on the input data based on the filter kernel determined through training.

The fusion layer 1210 includes a current frame (X _t ), data output through the gating process, that is, a weighted prediction frame (X' _{t_pred} ), a weighted prediction output frame (Y' _{t_pred} ), and a weighted prediction feature. Fuse the map (S' _{t_pred} ).

First, the current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), and the weighted prediction output frame (Y' _{t_pred} ) are concatenated 1212 and then input to the first convolutional layer 1214 . . Concatenation may mean a process of combining the current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), and the weighted prediction output frame (Y' _{t_pred} ) in the channel direction.

Data obtained as a result of the concatenation 1212 are convolutionally processed in the first convolutional layer 1214 . 3x3x1 displayed in the first convolution layer 1214 exemplifies convolution processing on input data using one filter kernel having a size of 3x3. As a result of the convolution process, one feature map is generated by one filter kernel.

The result of concatenating the current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), and the weighted prediction output frame (Y' _{t_pred} ) is input to the first convolutional layer 1214 and weighted prediction features separately The map S' _{t_pred} is input to the second convolutional layer 1216 . The weighted prediction feature map (S' _{t_pred} ) is convolutionally processed in the second convolutional layer 1216 . 3x3x1 displayed in the second convolution layer 1216 exemplifies convolution processing on input data using one filter kernel having a size of 3x3. As a result of the convolution process, one feature map is generated by one filter kernel.

Data output from the first convolution layer 1214 and data output from the second convolution layer 1216 are concatenated 1218 and then sequentially processed by a plurality of third convolution layers 1230 .

In the fusion layer 1210, unlike the current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), and the weighted prediction output frame (Y' _{t_pred} ), only the weighted prediction feature map (S' _{t_pred} ) is distinguished and the first 2 The reason for input to the convolution layer 1216 is the domain of the weighted prediction feature map (S' _{t_pred} ) and the current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), and the weighted prediction output frame (Y). This is because the domains of ' _{t_pred} ) are different. The weighted prediction feature map (S' _{t_pred} ) is data of the feature domain obtained in the process of processing the frame, but the current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), and the weighted prediction output frame (Y' _{t_pred} ) ) is image data corresponding to the processing target or image data obtained as a result of processing, so they are concatenated after distinguishing them. That is, the first convolutional layer 1214 and the second convolutional layer 1216 are a current frame (X _t ), a weighted prediction frame (X' _{t_pred} ), a weighted prediction output frame (Y' _{t_pred} ), and a weight, respectively. It may serve to match the domain of the applied prediction feature map (S' _{t_pred} ).

The data output from the first convolutional layer 1214 and the data output from the second convolution layer 1216 are concatenated 1218 and sequentially processed by a plurality of third convolutional layers 1230 to form an intermediate An output frame Y _{t_int} may be obtained. As shown in FIG. 12 , an intermediate output frame Y _{t_int} is output from the last layer 1234 of the plurality of third convolutional layers 1230 , and an intermediate feature from the previous layer 1232 of the last layer 1234 . A map (S _{t_int} ) is output. Although FIG. 12 illustrates that the last layer 1234 is positioned after the previous layer 1232 , one or more convolutional layers may be positioned between the previous layer 1232 and the last layer 1234 .

3x3x1 displayed in the third convolutional layers 1230 exemplifies that convolution processing is performed on input data using one filter kernel having a size of 3x3. As a result of the convolution process, one feature map or one output frame may be generated by one filter kernel.

The intermediate feature map S _{t_int} and the intermediate output frame Y _{t_int} output from the plurality of third convolutional layers 1230 are input to the next sub-neural network.

When the neural network includes only one sub-neural network, the current output frame Y _t is output from the last layer 1234 of the plurality of third convolutional layers 1230 , and the previous layer 1232 of the last layer 1234 is output. ), the current feature map (S _t ) is output.

The current output frame (Y _t ) and the current feature map (S _t ) may be used in the processing of the next frame.

The structure of the first sub-neural network 1200 shown in FIG. 12 is an example, and the number of convolution layers, the size of filter kernels, and the number of filter kernels included in the first sub-neural network 1200 vary depending on the implementation method. can be changed to

13 is a diagram illustrating the structure of the last sub-neural network 1300 included in the neural network.

Like the first sub-neural network 1200 , the last sub-neural network 1300 also includes a fusion layer 1310 including a first convolutional layer 1314 and a second convolutional layer 1316 , and a plurality of third convolutional layers. (1330). In the convolution layer, a convolution operation is performed on the input data based on the filter kernel determined through training.

The fusion layer 1310 includes the current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), the intermediate output frame (Y _{t_int} ) output from the previous sub-neural network, and the intermediate feature map (S _{t_int} ) output from the previous sub-neural network. to fuse

First, the current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), and the intermediate output frame (Y _{t_int} ) are concatenated 1312 and then input to the first convolutional layer 1314 .

Data obtained as a result of the concatenation 1312 are convolutionally processed in the first convolution layer 1314 . 3x3x1 displayed in the first convolution layer 1314 exemplifies convolution processing on input data using one filter kernel having a size of 3x3. As a result of the convolution process, one feature map is generated by one filter kernel.

The result of concatenating the current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), and the intermediate output frame (Y _{t_int} ) is input to the first convolutional layer 1314 and the intermediate feature map (S _{t_int} ) separately is input to the second convolutional layer 1316 . As described above, the intermediate feature map (S _{t_int} ) is convolutionally processed in the second convolutional layer 1316 . 3x3x1 displayed in the second convolution layer 1316 exemplifies convolution processing on input data using one filter kernel having a size of 3x3. As a result of the convolution process, one feature map is generated by one filter kernel.

As described above, the first convolutional layer 1314 and the second convolutional layer 1316 in the fusion layer 1310 are the current frame (X _t ), the weighted prediction frame (X' _{t_pred} ), and the intermediate output frame, respectively. (Y _{t_int} ) and the intermediate feature map (S _{t_int} ) may serve to match the domains.

Data output from the first convolution layer 1314 and data output from the second convolution layer 1316 are concatenated 1318 and then sequentially processed by a plurality of third convolution layers 1330 .

The data output from the first convolution layer 1314 and the data output from the second convolution layer 1316 are concatenated and then sequentially processed by a plurality of third convolution layers 1330 to provide a current output frame ( Y _t ) can be obtained.

As shown in FIG. 13 , the current output frame Y _t is output from the last layer 1334 of the plurality of third convolutional layers 1330 , and the current feature from the previous layer 1332 of the last layer 1334 . A map S _t is output. Although FIG. 13 illustrates that the last layer 1334 is positioned after the previous layer 1332 , one or more convolutional layers may be positioned between the previous layer 1332 and the last layer 1334 .

If the sub neural network 1300 shown in FIG. 13 is not the last sub neural network, an intermediate output frame Y _{t_int} is output from the last layer 1334 of the plurality of third convolutional layers 1330, and the last layer 1334 An intermediate feature map S _{t_int} may be output from the previous layer 1332 of . The output intermediate output frame (Y _{t_int} ) and intermediate feature map (S _{t_int} ) may be input to the next sub-neural network.

3x3x1 displayed in the third convolutional layers 1330 exemplifies convolution processing on input data using one filter kernel having a size of 3x3. As a result of the convolution process, one feature map or one output frame may be generated by one filter kernel.

The structure of the sub neural network 1300 shown in FIG. 13 is an example, and the number of convolution layers, the size of filter kernels, and the number of filter kernels included in the sub neural network 1300 may be variously changed according to an implementation method. have.

14 is a diagram illustrating an application example of an image processing method according to an embodiment.

The application example shown in FIG. 14 shows a process of obtaining output frames having a resolution greater than the resolution of the input frames through image processing of the input frames.

The video neural network (VNN) 1400 of FIG. 14 corresponds to the neural network 250 shown in FIG. 2 . It is assumed that the above-described motion prediction process, motion compensation process, weight derivation process, and gating process are performed before frames are input to the VNN 1400 .

As the first frame 1412 is processed by the VNN 1400 , a first output frame 1432 greater than the resolution of the first frame 1412 is obtained. The first frame 1412 and the first output frame 1432 are input to the VNN 1400 together with the second frame 1414 , and as a result of processing by the VNN 1400 , the resolution of the second frame 1414 is greater than that of the second frame 1414 . A second output frame 1434 with resolution is obtained. The second frame 1414 and the second output frame 1434 are input to the VNN 1400 together with the third frame 1416 , and as a result of processing by the VNN 1400 , the resolution of the third frame 1416 is greater than that of the third frame 1416 . A third output frame 1436 with resolution is obtained.

The application example shown in FIG. 14 may be useful when it is desired to increase the resolution of frames received from a server or the like. The server may transmit a small bitrate to the image processing apparatus 100 by encoding the frames of the small resolution, and the image processing apparatus 100 processes the frames of the small resolution obtained through decoding to output a larger resolution. frames can be obtained.

15 is a diagram illustrating an application example of an image processing method according to another embodiment.

The application example shown in FIG. 15 shows a process of obtaining one output frame in which the characteristics of the input frames are combined through image processing on the input frames.

As described above, it is assumed that the aforementioned motion prediction process, motion compensation process, weight derivation process, and gating process are performed before frames are input to the VNN 1500 .

The first frame 1512 is input to the VNN 1500 , and the processing result of the first frame 1512 by the VNN 1500 is the VNN 1500 together with the first frame 1512 and the second frame 1514 . is entered as Then, the processing result by the VNN 1500 is again input to the VNN 1500 together with the second frame 1514 and the third frame 1516 . As a result of processing by the VNN 1500 , an output frame 1530 in which all characteristics of the first frame 1512 , the second frame 1514 , and the third frame 1516 are reflected may be obtained.

The application example shown in FIG. 15 may be useful for improving the dynamic range of frames. For example, if one frame is photographed with a long exposure time and the other frame is photographed with a short exposure time, an output frame having a high dynamic range may be obtained by including the characteristics of both frames.

16 is a diagram illustrating an application example of an image processing method according to another embodiment.

An application example illustrated in FIG. 16 is considered when the image processing apparatus 100 operates as a server or an image provider. In general, a server encodes an image and transmits it to a terminal device, and the terminal device decodes a bitstream received from the server to restore the image. The image processing apparatus 100 may also use a frame encoded by the encoder 120 and then decoded by the decoder 140 when processing frames to compensate for a loss occurring in the encoding/decoding process.

Specifically, the image processing apparatus 100 processes the first frame 1612 based on the VNN 1600 to obtain a first output frame A (not shown). A first bitstream is generated through encoding of the first output frame A, and the first output frame A is restored through decoding of the first bitstream. The image processing apparatus 100 processes the first output frame A with the VNN 1600 to obtain a first output frame B (not shown).

The first frame 1612 , the first output frame A and the first output frame B are input to the VNN 1600 together with the second frame 1614 . In the above-described embodiments, one output frame is input to the VNN 1600 together with the next frame, but in the application example shown in FIG. 16 , two output frames are input to the VNN 1600 together with the next frame. A motion compensation process and a gating process may be applied to both output frames before being input to the VNN 1600 .

The second frame 1614 , the first frame 1612 , the first output frame A and the first output frame B are processed by the VNN 1600 to obtain a second output frame A. A second bitstream is generated through encoding of the second output frame A, and the second output frame A is restored through decoding of the second bitstream. The image processing apparatus 100 processes the second output frame A and the first output frame B with the VNN 1600 to obtain a second output frame B (not shown). Although not shown, the image processing apparatus 100 processes the first output frame A restored through decoding together with the second output frame A and the first output frame B with the VNN 1600 to obtain the second output frame (A). 2 output frames B (not shown) may be obtained.

The second frame 1614 , the second output frame A and the second output frame B are input to the VNN 1600 together with the third frame 1616 . A third frame 1616 , a second frame 1614 , a second output frame A and a second output frame B may be processed by the VNN 1600 to obtain a third output frame A .

17 is a flowchart of a multi-frame processing method according to an embodiment.

In operation S1710, the image processing apparatus 100 identifies a prediction sample corresponding to the current sample of the current frame from the previous frame. In order to identify a prediction sample, motion prediction may be performed on the current frame and the previous frame. As described above, a convolution operation may be performed on the current frame and the previous frame to identify the prediction sample.

In operation S1720, the image processing apparatus 100 generates a prediction frame of the current frame by changing the sample value of the collocated sample of the previous frame according to the sample value of the prediction sample. A prediction frame may be generated through a convolution operation on a previous frame based on filter kernels corresponding to motion vectors.

In operation S1730, the image processing apparatus 100 derives a weight by comparing the sample value of the current sample with the sample value of the prediction sample. The image processing apparatus 100 determines a weight to be small as the difference value between the sample value of the current sample and the sample value of the prediction sample is large, and determines the weight to be large as the difference value between the sample value of the current sample and the sample value of the prediction sample is small. can

In operation S1740, the image processing apparatus 100 applies a weight to the collocated samples of the prediction frame. The image processing apparatus 100 may multiply the collocated sample of the prediction frame by a weight.

In operation S1750, the image processing apparatus 100 processes the weighted prediction frame and the current frame through a neural network including a convolutional layer to obtain a current output frame.

Meanwhile, the above-described embodiments of the present disclosure can be written as a program that can be executed on a computer, and the written program can be stored in a device-readable storage medium.

The device-readable storage medium may be provided in the form of a non-transitory 0 storage medium. Here, 'non-transitory storage medium' is a tangible device and only means that it does not contain a signal (eg, electromagnetic wave). It does not distinguish the case where it is stored as For example, the 'non-transitory storage medium' may include a buffer in which data is temporarily stored.

According to an embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store™) or on two user devices (eg, It can be distributed (eg downloaded or uploaded) directly or online between smartphones (eg: smartphones). In the case of online distribution, at least a portion of the computer program product (eg, a downloadable app) is stored at least on a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server. It may be temporarily stored or temporarily created.

In the above, the technical idea of the present disclosure has been described in detail with reference to preferred embodiments, but the technical idea of the present disclosure is not limited to the above embodiments, and those of ordinary skill in the art within the scope of the technical spirit of the present disclosure Various modifications and changes are possible by the person.

Claims (16)

a memory storing one or more instructions; and
a processor executing the one or more instructions stored in the memory;
The processor is
Identifies a prediction sample corresponding to the current sample of the current frame from the previous frame,
A prediction frame of the current frame is generated by changing a sample value of a collocated sample of the previous frame according to a sample value of the prediction sample,
A weight is derived by comparing the sample value of the current sample with the sample value of the prediction sample,
applying the weight to the collocated samples of the prediction frame;
An image processing apparatus for obtaining a current reconstructed frame by processing the weighted prediction frame and the current frame through a neural network including a convolutional layer.
The method of claim 1,
The processor is
and a sample having a sample value most similar to the sample value of the current sample among the collocated sample of the previous frame and neighboring samples of the collocated sample as the prediction sample.
3. The method of claim 2,
The processor is
Convolving the current sample and neighboring samples of the current sample with a predetermined first filter kernel to obtain a sample value corresponding to the first filter kernel,
Convolving the collocated sample of the previous frame and neighboring samples of the collocated sample with a plurality of predetermined second filter kernels to obtain sample values corresponding to the plurality of second filter kernels;
identify a sample value most similar to a sample value corresponding to the first filter kernel from among sample values corresponding to the plurality of second filter kernels;
and determining, as the prediction sample, a sample corresponding to the identified sample value among the collocated sample of the previous frame and neighboring samples of the collocated sample.
4. The method of claim 3,
The first filter kernel is
A sample corresponding to the current sample has a preset first value, and other samples have a value of 0.
5. The method of claim 4,
The plurality of second filter kernels,
Any one sample has a preset second value, other samples have a value of 0,
The position of the one sample having the second value is different for each of the plurality of second filter kernels.
6. The method of claim 5,
and signs of the first preset value and the preset second value are opposite to each other.
4. The method of claim 3,
The processor is
Changing the sample value of the collocated sample by convolution processing the collocated sample of the previous frame and neighboring samples of the collocated sample with a third predetermined filter kernel,
In the third filter kernel, a sample corresponding to the prediction sample has a preset third value, and other samples have a value of 0.
According to claim 1,
The weight is in inverse proportion to a difference between a sample value of the current sample and a sample value of the prediction sample.
According to claim 1,
The processor is
Obtaining a previous output frame and a previous feature map output as a result of processing the previous frame by the neural network,
generating a prediction output frame and a prediction feature map by changing the sample values of collocated samples of the previous output frame and the previous feature map according to the positional relationship between the current sample and the prediction sample in the previous frame;
applying the weight to the collocated samples of the prediction output frame and the prediction feature map;
and inputting the weighted prediction output frame, the weighted prediction feature map, the weighted prediction frame, and the current frame to the neural network.
10. The method of claim 9,
The previous output frame is
A first previous output frame output from the neural network and a second previous output frame obtained as a result of processing the first previous output frame restored through encoding and decoding of the first previous output frame by the neural network , image processing device.
10. The method of claim 9,
The neural network includes a plurality of sub-neural networks including a first convolutional layer, a second convolutional layer, and a plurality of third convolutional layers,
The first convolution layer of the first sub-neural network performs convolution processing on a result of concatenating the weighted prediction output frame, the weighted prediction frame, and the current frame,
The second convolutional layer of the first sub-neural network convolves the weighted prediction feature map,
The plurality of third convolutional layers of the first sub-neural network sequentially concatenates the result of concatenating the feature map output from the first convolution layer and the feature map output from the second convolution layer, image processing device.
12. The method of claim 11,
A first convolutional layer of a sub-neural network other than the first sub-neural network is a concatenated result of the weighted prediction frame, the current frame, and intermediate reconstructed frames output from the previous sub-neural network. Root processing,
The second convolutional layer of the sub-neural network other than the first sub-neural network convolves an intermediate feature map output from the previous sub-neural network,
The plurality of third convolutional layers of the sub-neural network other than the first sub-neural network sequentially concatenates the result of concatenating the feature map output from the first convolution layer and the feature map output from the second convolution layer. A solution processing, image processing device.
According to claim 1,
The processor is
An image processing apparatus for transmitting a bitstream generated through encoding of the current output frame to a terminal device.
According to claim 1,
The processor is
An image processing apparatus for reproducing the current output frame through a display.
In the multi-frame processing method by an image processing apparatus,
identifying a prediction sample corresponding to a current sample of a current frame from a previous frame;
generating a prediction frame of the current frame by changing a sample value of a collocated sample of the previous frame according to a sample value of the prediction sample;
deriving a weight by comparing the sample value of the current sample with the sample value of the prediction sample;
applying the weight to collocated samples of the prediction frame; and
A method of processing multi-frames, comprising: processing the weighted prediction frame and the current frame through a neural network including a convolutional layer to obtain a current reconstructed frame.
A computer-readable recording medium recording a program for executing the method of claim 15 in combination with hardware.

Images