KR20220165121A - Method and apparatus for progressive image resolution improvement using neural ordinary differential equation, and image super-resolution method using the smae - Google Patents

Abstract

A gradual image resolution enhancement method and apparatus, and an image super-resolution method using the same are disclosed, wherein by expressing the process of gradually changing a low-resolution image into a high-resolution image with an ordinary differential equation including a neural network, the super-resolution problem can be solved by making the same an initial value problem. The gradual image resolution enhancement method comprises the steps of: (a) starting from a first input image by upscaling a first input image, inputting an (N-1)^th (N is a natural number greater than or equal to 2) input image into a neural network, applying a scale factor, and outputting an (N-1)^th residual image from the neural network; (b) generating an N^th input image with the sum of the (N-1)^th residual image and the (N-1)^th input image, wherein the N^th input image is passed as input to the neural network; and (c) repeating the steps (a) and (b) while gradually decreasing the size of the scale factor by a preset size until the scale factor becomes 1. When the scale factor of 1 is applied, a final output image is generated as the sum of a first input image and an accumulated calculation value of image residuals between the (N-1)^th residual image and the N^th residual image.

Description

Gradual image resolution enhancement method and apparatus using ordinary differential equations including neural networks, and image super-resolution method using the same

The present invention relates to an image processing technology for super-resolution, and more particularly, by expressing the process of gradually changing a low-resolution image into a high-resolution image as an ordinary differential equation including a neural network, the super-resolution problem is solved by making it an initial value problem. It relates to a gradual image resolution enhancement method and apparatus and an image super-resolution method using the same.

With the recent spread of ultra-high definition (HUD) displays, research on super resolution (SR) technology converts conventional low resolution (LR) images into relatively high resolution (HR) images. Interest is growing.

Existing super-resolution techniques include a polynomial-based interpolation method such as bicubic interpolation or a local patch-based super-resolution technique using linear mapping.

Recently, one of the studies on signal image super resolution (SISR) is to learn mapping functions between LR image match and HR image patch. In these previous studies, the results of simplifying the optimization process of convolutional neural networks were obtained by applying different types of skip connections.

On the other hand, super-resolution technology that uses a mapping function to directly restore a high-resolution patch from a low-resolution patch can generate a high-quality, high-resolution image with relatively little complexity and computation. It is difficult to implement a nonlinear super-resolution model.

In order to solve the problem in super-resolution using such a mapping function, various studies on super-resolution algorithms based on deep learning using a convolutional neural network (CNN) have recently been conducted. Representative existing studies include VDSR (very-deep supper-resolution), one of the deep learning algorithms for SISR, residual dense network (RDN), and SRCNN (super-resolution convolution neural network).

However, existing deep learning-based super-resolution techniques have a structure of upsampling the resolution in the second half of the network, and since this structure performs mapping from low resolution to high resolution at once, especially in high magnification or high frequency areas that predict a lot of information In this many images, the peak signal-to-noise ratio (PSNR) performance is poor, the intermediate process of super-resolution is unknown, and there is a limit to super-resolution only at a pre-determined magnification.

To provide a deep learning-based super-resolution technology that can overcome the limitations of the above-mentioned prior art of the present invention, an object of the present invention is to use an ordinary differential equation including a neural network, that is, NODE (Neural Ordinary Differential Equation) It is to provide a method of improving the resolution of an image and a method of super-resolution a device and an image.

Another object of the present invention is a gradual image resolution that can observe or analyze the process of changing an image from low resolution to high resolution by analyzing intermediate images in the process of solving an ordinary differential equation, rather than simply doing super resolution in the super resolution of an image. An enhancement method and apparatus and an image super-resolution method are provided.

Another object of the present invention is to provide an image super-resolution device using the above-described gradual image resolution enhancement method or image super-resolution method.

In the gradual image resolution enhancement method according to an aspect of the present invention for solving the above technical problem, starting from the first input image obtained by upscaling the first input image, the N-1th (N is a natural number greater than or equal to 2) input (a) outputting an N-1 th residual image from the neural network by inputting the image to the neural network and applying a scale factor; (b) generating an N-th input image from the sum of the N-1-th residual image and the N-1-th input image, wherein the N-th input image is transmitted as an input of the neural network; and (c) repeating steps (a) and (b) while gradually decreasing the size of the scale factor by a preset size until the scale factor becomes 1. When a scale factor of 1 is applied, a final output image is generated as a sum of a cumulative calculation value of image residuals between the N−1 th residual image and the N th residual image and the first input image.

In one embodiment, the scale factor may be a rational number value greater than one.

In an embodiment, the scale factor is an input added to the neural network, and may be a value that sequentially decreases in a geometrical, linear, geometrical, or non-linear manner for each repetition of the neural network.

In one embodiment, the cumulatively calculated value may be modeled as an ordinary differential equation (ODE) represented by Equation 4 below:

[Equation 4]

은 상기 최종 출력 이미지를,

는 초기 스케일 팩터(t_o)가 적용된 상기 제1 번째 입력 이미지를,

는 상기 뉴럴 네트워크의 입력 이미지에 점진적으로 감소되는 스케일 팩터(t)가 적용된 이미지를, f()dt는 스케일 팩터(t)가 추가 입력되는 상기 뉴럴 네트워크의 입력 이미지와 출력 이미지 간의 이미지 잔차에 대한 상미분 방정식을,

는 상기 뉴럴 네트워크의 트레이닝 가능한 파라미터를 각각 나타낸다.In Equation 4,

is the final output image,

Is the first input image to which the initial scale factor (t _o ) is applied,

is an image to which a gradually reduced scale factor (t) is applied to the input image of the neural network, and f()dt is an image residual between the input image and the output image of the neural network to which the scale factor (t) is additionally input. an ordinary differential equation,

denotes trainable parameters of the neural network, respectively.

는 원본 이미지(I_HR)을 t배 줄이고, 이것을 다시 t배 키운 이미지가 된다. 여기서, 이미지를 줄이고 키우는 방식 즉, 업스케일링(upscaling)이나 다운스케일링(downscaling) 방법은 바이큐빅(bicubic) 방법이나 다른 선형 보간 방법이 사용될 수 있다.In one embodiment, the

is reduced by t times the original image (I_HR), and becomes an image enlarged by t times again. Here, a bicubic method or another linear interpolation method may be used as a method of reducing and enlarging an image, that is, an upscaling or downscaling method.

In one embodiment, the cumulatively calculated value may be calculated numerically by an ODE solver. The ODE solver may include one or more selected from the Euler method, the Runge-Kutta method, the Heun method, the midpoint method, and the adaptive Runge-Kutta method.

In one embodiment, a neural network can be trained with parameters to minimize the loss function represented by Equation 6:

[Equation 6]

는 파라미터

를 훈련하여 손실함수를 최소화하는 수학식을,

은 상대적 고해상도(HR: high resolution)의 훈련용 원본 이미지를, 그리고 ODESolove()는 ODE 솔버 함수를 각각 나타낸다.In Equation 6,

is the parameter

The equation for minimizing the loss function by training

represents the original training image of relative high resolution (HR), and ODESolove() represents the ODE solver function, respectively.

In one embodiment, the progressive image resolution enhancement method may further include updating parameters of the neural network through backpropagation.

A gradual image resolution enhancement device according to another aspect of the present invention for solving the above technical problem includes a processor; memory electronically connected to the processor; and instructions stored in the memory, which, when executed by the processor, start with the first input image obtained by upscaling the first input image by the processor, and start with the N-1th (N is a natural number greater than or equal to 2) input image. (a) outputting an N-1 th residual image from the neural network by inputting to the neural network and applying a scale factor; (b) generating an N-th input image from the sum of the N-1-th residual image and the N-1-th input image, where the N-th input image is transmitted as an input of the neural network; and (c) repeatedly performing steps (a) and (b) while gradually decreasing the size of the scale factor by a preset size until the scale factor becomes 1; and a step (d) of generating a total output image as a sum of a value obtained by accumulatively calculating image residuals between the N-1 th residual image and the N th input image when the scale factor is applied as 1 and the first input image. let it do

In the image super-resolution method according to another aspect of the present invention for solving the above technical problem, in the image super-resolution method performed by a processor, an output image of a neural network to which an input image is input is converted into an input image of the neural network. re-entering as; sequentially providing scale factors, the sizes of which are gradually set to be different, to the neural network whenever an input image is input to the neural network; calculating an image residual between an input image and an output image of the neural network by using an ordinary differential equation for each repetitive operation of the neural network and accumulating or updating the calculated values; and generating a relatively high-resolution final output image by summing the final image residual obtained in the updating step and the initial input image.

In one embodiment, the providing step additionally inputs a scale factor to the first layer of the neural network in addition to the input image.

In one embodiment, the inputting and the providing may include adding padding to an edge of the input image in response to a scale factor having a size of a rational number greater than 1 based on the input image, and displaying the input image to which the padding is added. Center cropping may be performed to correspond to the original size of the input image around the center pixel.

In one embodiment, the neural network may include a residual dense network (RDN). The RDN has a form in which the last upsampling layer is omitted, so that the input resolution and the output resolution may be identical.

In one embodiment, the image super-resolution method may further include training a neural network. The training step is a single training process of estimating image details through an image residual between an input image and an output image of the neural network through an ordinary differential equation, updating or accumulating the image details each time the neural network is performed, and adding them to the first input image. may be performed for different scale factor sets so that the preset loss function becomes a minimum value or less than a preset reference value.

According to the present invention, by adopting a method of gradually super-resolution an image using an ordinary differential equation that includes a neural network, that is, a Neural Ordinary Differential Equation (NODE), the neural network learns information that can make a low resolution into a high resolution , Ordinary differential equations can express the relationship between images of various resolutions, thereby outperforming conventional methods using convolutional neural networks (CNNs) in super-resolution tasks, and super-resolution There are advantages to doing it.

In addition, according to the present invention, it can be effectively used when an image is to be enlarged at a desired magnification on a television (TV), smartphone, etc., and image editors such as Adobe Photoshop, Adobe Premiere Pro, etc. It can be used effectively when you want to grow images or videos in video editing application software. In addition, it is possible to effectively provide a super-resolution method required by software such as an image editor, a video editor, or a closed circuit television (CCTV) operation/management/control program.

1 is a flowchart of a method for gradually increasing image resolution according to an embodiment of the present invention.
FIG. 2 is a conceptual diagram illustrating the gradual image resolution enhancement method of FIG. 1 .
FIG. 3 is a diagram illustrating input and output of a neural network in the gradual image resolution enhancement method of FIG. 2 .
4 is a block diagram of an apparatus for gradually increasing image resolution according to another embodiment of the present invention.
5 is a block diagram of an image super-resolution device according to another embodiment of the present invention.
FIG. 6 is a block diagram of a detailed configuration of a pre-processing unit that can be employed in the image super-resolution device of FIG. 5 .
FIG. 7 is a block diagram of a detailed configuration of a training processing unit that can be employed in the image super-resolution device of FIG. 5 .
FIG. 8 is a flowchart for explaining an image super-resolution method employable in the image super-resolution apparatus of FIG. 5 .
9 is a diagram showing a comparison of super-resolution results of the super-resolution method (NODE-RDN) of this embodiment and the super-resolution methods (DRRN, SRFBN) of the comparative example.
10 is a diagram showing a comparison of another super-resolution result of a super-resolution method (NODE-RDN) of this embodiment and super-resolution methods (DRRN, SRFBN) of a comparative example.
11 is a diagram showing a comparison of super-resolution results of a super-resolution method (NODE-RDN) of this embodiment and a super-resolution method (Meta-RDN) of a comparative example.
12 is a graph showing the performance (PSNR) evaluated while changing the scale factor in the super-resolution methods (NODE-RDN, NODE-VDSR) of the present embodiment.
13 is a graph for explaining a principle of setting a scale factor that can be employed in the image super-resolution method of the present embodiment based on the strength of intermediate image residuals.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals have been used for like elements throughout the description of each figure.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term “and/or” includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this application, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart of a method for gradually increasing image resolution according to an embodiment of the present invention.

Referring to FIG. 1 , in step (a), starting from a first input image obtained by upscaling the first input image, the gradual image resolution enhancement method starts with an N-1th (N is a natural number greater than or equal to 2) input image. is input to the neural network and a scale factor is applied to output an N-1 th residual image from the neural network (S11).

The first input image is upsampled to correspond to the size (for example, M times) when the output image has the first resolution, considering the output image of the second resolution higher than the first resolution of the original image having the first resolution. Refers to an image converted through upsampling or upscaling.

In addition, the first input image is first downscaled to a predetermined size (eg, 1/M times) of the first relatively high-resolution original image in the training mode of the neural network, and then again to a predetermined size (eg, M times). It may correspond to an image subjected to upscaling.

Downscaling or upscaling for the first input image may be processed through an interpolation method or a super-resolution kernel for estimating a value between points in an image given discontinuously. The interpolation method or super-resolution kernel may include linear interpolation, bicubic interpolation, bilinear interpolation, bicubic interpolation, and the like.

Next, in step (b), an N-th input image is obtained as a sum of the N-1 th residual image and the N-1 th input image (S12). Here, the Nth input image is transmitted as an input of the neural network.

Next, in step (c), steps (a) and (b) are repeatedly performed while gradually decreasing the size of the scale factor by a preset size until the scale factor becomes 1. To this end, it is determined whether the current scale factor input as a gradually reduced value is 1 (S13), and if the current scale factor is not 1, the current output image of the neural network is converted into a new input image of the neural network again. While transferring, the process of calculating the difference between the previous input image and the current output image of the neural network, that is, the image residual, by an ordinary differential equation is repeated, and this iterative process can be performed until the scale factor becomes 1 (S14). .

An ordinary differential equation refers to a differential equation in which an unknown function and a dependent variable are functions having one independent variable. In this embodiment, the unknown function corresponds to a neural network, and the dependent variable corresponds to a scale factor and parameters of the neural network. .

A parameter of a neural network represents a weight that is multiplied by an input value passed from a previous layer to a current layer among layers of the neural network. can be set differently. The activation function refers to a function that sets a predetermined threshold and changes an output value, and includes a ReLU function, a sigmoid function, and the like.

In this embodiment, the final output image of the processor implementing the gradual image resolution enhancement method or the gradual image resolution enhancement device having such a processor is basically the N-1th residual image and the Nth residual image when a scale factor of 1 is applied. It may be generated by summing the cumulative calculation value of the image residuals between the th residual images and the first input image.

Also, in this embodiment, the value of the initial scale factor is a rational number greater than 1. This scale factor is an input added to the neural network in addition to the image input, and may be a value that sequentially decreases in a geometrical, linear, geometrical, or nonlinear manner for each repetitive operation of the neural network.

In addition, the scale factor is the type or content of the original image to be improved in resolution, for example, in the case of a first image including a relatively high frequency region, such as an eye with an iris of a human face, such as a cheek of a human face. Depending on the case of the second image including a relatively large number of low-frequency regions, it may be designed to use initial scale factors of different sizes or to use scale factors that gradually decrease at different intervals.

In addition, N (a natural number of 2 or more) is determined according to the number determined by the interval between the largest scale factor value and 1. The largest scale factor may be designed according to the type and contents of the target original image for which image resolution is to be improved, for example, a ratio of a high-frequency region or a low-frequency region. In particular, in this embodiment, weights or parameters of the neural network may be optimized by performing learning based on several scale factor sets according to the type of original image through deep learning.

FIG. 2 is a conceptual diagram illustrating the gradual image resolution enhancement method of FIG. 1 . FIG. 3 is a diagram illustrating input and output of a neural network in the gradual image resolution enhancement method of FIG. 2 .

Referring to FIG. 2, the gradual image resolution improvement method is a gradual super-resolution process in which a super resolution (SR) problem is made into an initial value problem and solved by an ordinary differential equation (ODE). Formulate and use it to implement image super-resolution based on neural networks.

That is, for the implementation of image super-resolution, a model using an iterative multi-stage based neural network, a preprocessed initial input image, and a minimum image difference in a continuous SR process based on the initial input image and the neural network, that is, Prepare a super-resolution model.

More specifically, first, a neural network using an iterative multi-stage is prepared.

The relationship between an input image and an output image of a neural network based on iterative multiple stages can be expressed as Equation 1 below.

은 반복적으로 재차 정의되는 중간 이미지들(n<N인 경우) 또는 최종 이미지(n=N인 경우)를,

은 현재의 중간 이미지나 최종 이미지 직전의 이전 이미지 또는 최초 이미지(n=2인 경우)를,

()은 이전 이미지에 적용되는 초해상도 네크워크 또는 뉴럴 네트워크를, n은 반복 단계를, 그리고 N은 반복 횟수를 각각 나타낸다.In Equation 1,

Is the intermediate images (in the case of n < N) or the final image (in the case of n = N) that are repeatedly redefined,

is the current intermediate image or the previous image immediately before the final image or the first image (in the case of n = 2),

() represents the super-resolution network or neural network applied to the previous image, n represents the iteration step, and N represents the number of iterations, respectively.

를 뉴럴 네트워크(f)에 제1 스케일 팩터(t0)와 함께 입력하고, 뉴럴 네트워크에서 나온 잔차 이미지를 제1 번째 입력 이미지와 합한 제1 출력 이미지

를 뉴럴 네트워크의 제2 번째 입력 이미지로 입력할 수 있다.That is, in this embodiment, the first input image

is input into the neural network (f) together with the first scale factor (t0), and the first output image is obtained by adding the residual image from the neural network to the first input image.

may be input as the second input image of the neural network.

와 제N-1 스케일 팩터를 입력으로 받고, 제N-1 번째 입력 이미지

와 제N-1 번째 잔차 이미지에 기반한 제N 번째 출력 이미지

를 출력할 수 있다.As such, the repetitive multi-stage based neural network according to the present embodiment is a neural network or neural network model having a first merging unit at an output stage, and is an N-1th input image.

and the N-1th scale factor as input, and the N-1th input image

and the Nth output image based on the N-1th residual image.

can output

를 바이큐빅(bicubic) 등의 기존의 SR 커널을 이용하여 제1 스케일 팩터(예컨대, 1/t)로 다운스케일한 버전일 수 있다. 그리고 최초 입력 이미지 즉, 제1 번째 입력 이미지

는, 원본 이미지를 SR 커널을 이용하여 제2 스케일 팩터(예컨대, t)로 업스케일한 것으로 정의될 수 있다.Next, a first input image preprocessed from the original image is prepared. The original image is, in the training mode of the neural network, the original original image or the ground-truth clean image.

may be a downscaled version with a first scale factor (eg, 1/t) using an existing SR kernel such as bicubic. And the first input image, that is, the first input image

, may be defined as up-scaling the original image with a second scale factor (eg, t) using the SR kernel.

와 제1 번째 입력 이미지는 동일한 공간 해상도(spatial resolution)를 가진다. 깨끗한 원본 이미지

는 상대적인 고해상도(high resolution, HR) 이미지에 대응하고, 제1 번째 입력 이미지는 상대적인 저해상도(low resolution, LR) 이미지에 대응될 수 있다.clean original image here

and the first input image have the same spatial resolution. clean original image

may correspond to a relatively high resolution (HR) image, and the first input image may correspond to a relatively low resolution (LR) image.

Next, a gradually changing scale factor is set for a continuous domain, and a neural network is designed to sequentially have one of these scale factors as an input.

와 출력 이미지

간의 이미지 잔차(image residual)를 뉴럴 네트워크

를 이용하여 모델화할 수 있다.To this end, we first design a neural network to calculate a high frequency image residual. Specifically, by setting the scale factor (t) to a value of a rational number greater than 1 (hereinafter referred to as the maximum scale factor value), and gradually decreasing or increasing two consecutive scale factors between the maximum scale factor value and 1 at a predetermined interval, The input image of the neural network received as input

and the output image

Neural network image residuals between livers

can be modeled using

The image residual can be expressed as Equation 2 below.

은 뉴럴 네트워크의 제N-1 번째 입력 이미지에 대응하고, 출력 이미지

은 뉴럴 네트워크의 제N 번째 출력 이미지에 대응될 수 있다. 이 경우, 제N-1 번째 입력 이미지는 통상 제N 번째 출력 이미지보다 더 많은 고주파 디테일(high frequency details)을 포함할 수 있다.In Equation 2, the input image

corresponds to the N-1th input image of the neural network, and the output image

may correspond to the Nth output image of the neural network. In this case, the N−1 th input image may include more high frequency details than the N th output image.

According to the configuration described above, a super-resolution model can be reconstructed with an ordinary differential equation including a neural network designed to calculate a high frequency image residual.

Here, the residual image of the neural network can be expressed as an ordinary differential equation such as Equation 3, similar to the case of the existing super-resolution architecture.

를,

는 뉴럴 네트워크(f)의 훈련가능한 파라미터를 각각 나타낸다.In Equation 3, the left side represents the residual image output from the neural network, and the right side represents the image of the neural network to which the scale factor (t) is applied.

cast,

denote trainable parameters of the neural network f, respectively.

That is, the gradual image resolution enhancement method of the present embodiment can effectively obtain a high-frequency image residual by setting a scale factor as an additional input in addition to an image input to the neural network and applying a gradually changing scale factor for each repeated neural network stage. It is configured so that

를 설계한다. 이를 나타내면, 다음의 수학식 4와 같다.Next, the final output image corresponds to the pristine original image through an iterative multi-stage super-resolution model.

to design Expressing this, it is the same as the following Equation 4.

는 뉴럴 네트워크의 파라미터를 나타낸다.In Equation 4,

represents the parameters of the neural network.

As can be seen from Equation 4, fine details predicted from the input image may be gradually summed through the neural network f, which is a model for predicting high-frequency details in the input image.

와 합침으로써 최종 출력 이미지

를 얻도록 구현될 수 있다.That is, the method of the present embodiment is an iterative multi-stage of a super-resolution model using scale factors that gradually change, for example, scale factors including discrete rational values at predetermined intervals between to and 1 (including to and 1). The image residuals obtained from are sequentially updated or accumulated, and the accumulated value, that is, the integral value of the image residuals is converted into a first input image.

final output image by merging with

can be implemented to obtain

The above-described super-resolution method may be expressed or implemented as in Equation 5 using an existing ordinary differential equation solver.

은 적분 간격을 각각 나타낸다.In Equation 5, ODESlove() is an ordinary differential equation (OLE) solver,

represents the integration interval, respectively.

As such, the method of the present embodiment does not need to consider the stop condition in the existing SR method, which occurs because the SR process of each CNN architecture is independent in the gradual SR process. In particular, in order to solve an ordinary differential equation that treats the SR problem as an initialization problem, that is, to infer the final application image, it is not necessary to consider the existing disconnection condition at all by rendering the output of the neural network for each operation or iteration.

가 미리 설정되는 기준값 이하가 되거나 훈련 결과의 우수한 성능에 대한 최소값이 되도록 파라미터를 최적화할 수 있다.In addition, the method of this embodiment trains the parameters of a super-resolution model including a neural network through various scale factor sets for a single initial input image, thereby training the parameters of the neural network.

Parameters can be optimized so that is less than or equal to a preset reference value or a minimum value for good performance of the training result.

를 학습할 수 있다. 이를 수식으로 나타내면 다음의 수학식 6과 같다.For example, by using the L1 loss function, a deep neural network is trained to minimize the loss of the neural network that is added for various scale factors (t), and its parameters

can learn If this is expressed as a formula, it is as shown in Equation 6 below.

은 뉴럴 네트워크의 손실을 나타낸다.In Equation 6,

represents the loss of the neural network.

As described above, in the gradual image resolution enhancement method of the present embodiment, by minimizing the loss function, it is possible to train parameters of the neural network to more accurately predict image details added to the neural network.

Meanwhile, the parameters of the neural network of this embodiment can be optimized using an existing backpropagation method. In this case, it is also possible to use a gradually increasing scale factor to train the neural network through backpropagation.

4 is a block diagram of an apparatus for gradually increasing image resolution according to another embodiment of the present invention. 5 is a block diagram of a detailed configuration of a pre-processing unit that can be employed in the gradual image resolution enhancement device of FIG. 4 .

Referring to FIG. 4 , the gradual image resolution enhancement device may include a processor 100 and a memory 200 electronically connected to the processor. The memory 200 stores programs or instructions. The memory 200 may be integrally mounted in the processor 100 .

The processor 100 includes a pre-processing unit 10, a scale factor (SF) management unit 20, a neural network 30, a first merging unit 40, an iterative processing unit 50, and an image residual calculation unit 60. ), an accumulation unit 65 and a second merging unit 70.

In the test mode of the neural network 30, the pre-processing unit 10 upscales the original image I _o to enlarge the low-resolution original image to a predetermined size, and upscales the original input image ( Create I _a ). In addition, the preprocessor 10 generates an original training image (corresponding to I _o ) obtained by reducing the clean original image I _c to a predetermined size in the training mode of the neural network 30, and training An initial input image (I _a ) may be generated by up-scaling an original image for use in a predetermined scale.

In addition, the pre-processing unit 10 may include a downscaling unit 12 and an upscaling unit 14 as shown in FIG. 5 . The downscaling unit 12 and the upscaling unit 14 may include a kernel such as bicubic. The pre-processing unit 10 may be implemented as a separate image pre-processing device without being mounted on the gradual image resolution enhancing device.

Referring back to FIG. 4, the SF management unit 20 determines the maximum scale factor determined according to user input or set as default, and the interval between the scale factors determined according to user input or set as default. Manages at least one set of scale factors determined according to (s). The maximum scale factor may be a rational number greater than 1, and the intervals of the scale factors have a value corresponding to one slope in a linear form such as at equal intervals or correspond to a plurality of tangential slopes in a nonlinear form such as the same ratio. can have values of

In addition, the SF management unit 20 may sequentially input a scale factor whose size gradually decreases or increases whenever an input image is input to the neural network in the gradual image resolution enhancement device.

In addition, the SF management unit 20 may change the scale factor from 1 to 4 at an interval of 0.1 in the training mode of the neural network 30 and provide it as an input to each CNN architecture in the neural network 30.

The neural network 30 may be used as a backbone convolution neural network (CNN) architecture by slightly modifying an existing very-deep supper-resolution (VDSR) or residual dense network (RDN). In the case of using VDSR or RDN, the neural network 30 of this embodiment may be referred to as NODE-VDSR or NODE-RDN.

In the VDSR or RDN of the neural network 30, each CNN architecture is a first layer modified to receive a scale factor as an additional input (eg, the fourth channel) in addition to the image input (eg, the first to third channels). ) can be provided. That is, each CNN architecture may include an input channel of a scale factor in which pixel locations of a newly combined channel (fourth channel) are filled with scalar values.

In addition, the neural network 30 may be configured to have the same input resolution and output resolution by removing the last upsampling layer in the RDN. In this RDN, there is no addition of extra parameters except for the first layer.

In addition, in general CNN, the minimum batch size (mini-batch size) is basically 16 (200 × 200 patches), but in the neural network 30 of this embodiment, 8 (130 × 200 patches) can be used as the minimum batch size. can This may be useful where the memory of a graphics unit of a computing device is limited.

The first merging unit 40 generates an output image by combining the input image and the residual image generated by the neural network 30 .

The repetition processing unit 50 checks whether the SR process of the neural network 30 has been repeated up to the number of previously prepared or preset scale factors or determines whether the scale factor is 1, and determines whether the set number of times is not reached or the scale factor is If it is not 1, the output image is passed back to the next input image of the neural network.

The image residual calculation unit 60 calculates an image residual between an input image and an output image of the neural network 30 . That is, the image residual calculation unit 60 calculates an image residual between the previous input image or the N−1 th input image of the neural network 30 and the current input image or the N th input image. The image residual calculator 60 may be implemented as an ordinary differential equation solver.

As the ODE solver of the image residual calculator 60, the Python torchdiffeq library [Chen et al., 2018] employing Runge-Kutta (RK4) can be used. In this case, only 6 additional lines of code are required with PyTorch, so the design of the SR system is easy, and the system is lightweight, so it can be easily installed in a mobile device such as a smartphone.

The accumulator 65 updates or accumulates results of calculating image residuals between an input image and an output image of the neural network whenever the neural network is repeatedly operated. The value accumulated in the accumulator 65 may correspond to an integral value obtained by integrating an ordinary differential equation corresponding to the function of the image residual calculator 60 .

The second merging unit 70, according to the signal or information from the iteration processing unit 50, the accumulation unit ( 65) and the initial input image (I _b ) to create the final output image (I _b ).

According to this embodiment, when executed by the processor by the instructions, the instructions are the N-1th (N-1)th (N is a natural number greater than or equal to 2) input an input image to a neural network and apply a scale factor to output an N-1th residual image from the neural network.

Further, the instructions cause the processor to generate, in step (b), an N-th input image as a sum of the N-1 th residual image and the N-1 th input image, and the N-th input image of the neural network. to be passed as input.

Further, the instructions cause the processor to repeatedly perform steps (a) and (b) while gradually reducing the magnitude of the scale factor by a preset amount until the scale factor equals 1 in step (c).

Further, the instructions cause the processor to calculate, in step (d), the sum of the image residuals between the N-1 th residual image and the N th input image and the first input image when the scale factor is applied as 1. Create or print the final output image.

6 is a block diagram of an image super-resolution device according to another embodiment of the present invention. 7 is a block diagram of a detailed configuration of a training processing unit that can be employed in the image super-resolution apparatus of FIG. 6 .

Referring to FIG. 6 , the image super-resolution device may include a processor 100 and a memory 200 electronically connected to the processor. The memory 200 stores programs or instructions. The memory 200 may be integrally mounted in the processor 100 .

The processor 100 includes a preprocessing unit 10, a scale factor (SF) management unit 20, a neural network 30, a first merging unit 40, an iterative processing unit 50, an image residual calculation unit 60, an accumulation It may include a unit 65, a second merging unit 70, an evaluation unit 80, and a training processing unit 90.

A pre-processing unit 10, a scale factor (SF) management unit 20, a neural network 30, a first merging unit 40, an iterative processing unit 50, an image residual calculation unit 60, an accumulation unit 65, and Since the second merging unit 70 is substantially the same as the corresponding component of the gradual image resolution enhancement device described above with reference to FIGS. 4 and 5 , a detailed description thereof will be omitted.

The evaluation unit 80 compares the final output image output through the second merging unit 70 with a clean original image, or compares final output images generated according to different scale factor sets with each other, or compares the final output image with the first input image. The SR performance of the final output image may be evaluated by comparing the performance of the first output image to a predetermined reference value (including the performance of the comparative example). To this end, the evaluation unit 80 may use a final output image stored in the memory 200 or the database 210, a clean original image, an image of a comparative example, and the like. Evaluation results of the evaluation unit 80 may be transmitted to the training processing unit 90 .

The training processing unit 90 manages training of the network to minimize the L1 loss using an Adam optimizer or the like according to a training condition according to a preset training policy or user input. The training processing unit 90 may be configured to set an initial learning rate to 10 ⁻⁴ , reduce the rate by half for every 100k gradient update steps, and perform repeated training for a total of 600k times.

Also, as shown in FIG. 7, the training processing unit 90 may include a parameter management unit 92, a parameter comparison unit 94, a backpropagation control unit 96, and a reference value management unit 98. Also, the training processing unit 90 may be implemented to include the SF management unit 20 in its modified example.

Looking more specifically at the components that can be employed in the training processing unit 90, the parameter management unit 92 manages the parameters of the neural network 30. The parameter management unit 92 may transmit a scale factor set set according to a user input or stored in advance to the SR management unit 20 during network training using various scale factor sets.

The performance comparison unit 94 may compare SR performance by various scale factor sets during network training. The performance comparison unit 94 may mutually compare performance of each SR result through PSNR, SSIM, and the like, and extract optimal parameter information for a scale factor set with the best performance. The extracted optimum parameter information may be fed back to the parameter management unit 92.

The backpropagation control unit 96 may control the operation of the neural network to update the parameters of the neural network through backpropagation included as one of a user input setting or a predetermined optimization condition. Based on the parameter information obtained from the performance comparison unit 94, the backpropagation control unit 96 connects the connection parameters in the order described from the output layer of each CNN architecture of the neural parameters to the input layer through at least one hidden layer from the output layer. It can be operated as a node to update step by step.

In backpropagation, the differential value of the weight of the error used for parameter update can be expressed by the activation function value of the starting node, the activation function value of the destination node, and the actual parameter value. In addition, the parameter update through backpropagation may be repeatedly performed along with the SR process described above with reference to FIG. 1 based on the SR performance comparison result of the performance comparison unit 94 or the parameter information obtained from the performance comparison unit 94. have.

The reference value management unit 98 manages an optimal value of a parameter obtained by minimizing a loss function such as an L1 loss function or a reference value close to the optimal value. The reference value may correspond to a target value or a target set value required for the performance of the SR system when super-resolution of an original image is performed using the SR system using the neural ordinary differential equation of the present embodiment. The reference value management unit 98 may provide the performance comparison unit 94 with a reference value set by a user input or stored in advance.

FIG. 8 is a flowchart for explaining an image super-resolution method employable in the image super-resolution apparatus of FIG. 5 .

Referring to FIG. 8 , the image super-resolution method is an image super-resolution method performed by a processor. First, an output image of a neural network to which an input image is input is input again as an input image of the neural network (S81).

The neural network may include a residual dense network (RDN). The RDN has a form in which the last upsampling layer is omitted, so that the input resolution and the output resolution may be identical.

Next, whenever an input image is input to the neural network, scale factors having gradually different sizes are sequentially provided to the neural network (S82). This step (S82) may be configured to additionally input a scale factor to the first layer of the neural network in addition to the input image.

In the above two steps (S81 and S82), padding is added to the edge of the input image in response to a scale factor having a size of a rational number greater than 1 based on the input image, and the center pixel of the padded input image is centered. With , center cropping may be performed to correspond to the original size of the input image.

In addition, in the above two steps (S81 and S82), the neural network gradually decreases at preset intervals from the initial scale factor whose scale factor is a rational number greater than 1, and operates repeatedly until the scale factor finally becomes 1. It can (S83, S84).

Next, image residuals between an input image and an output image of the neural network are calculated using an ordinary differential equation for each repetitive operation of the neural network, and the calculated values are accumulated or updated (S85). Accumulation or update may be determined according to a method of separately storing each calculation result of an ordinary differential equation and then combining them, or combining a current calculation result with a previous calculation result and storing the result.

Next, a relatively high-resolution final output image is generated by adding the final image residual obtained through the accumulation or update process and the initial input image (S86). The final image residual may correspond to an integral value obtained by accumulating calculated values of an ordinary differential equation for image residuals between a previous residual image, that is, the N-1th residual image, and a current residual image, that is, the Nth residual image.

Next, super-resolution performance can be selectively evaluated (S87). Super-resolution performance evaluation may be performed by comparing comparison results between an original image or a first input image and a final output image, comparing final output images with different scale factor sets, and the like.

Also, if necessary, parameters of the neural network may be selectively updated through backpropagation (S88). Updating parameters through backpropagation may be performed in a training mode of a neural network.

In this case, in the training mode, the processor estimates image detail through an image residual between an input image and an output image of the neural network through an ordinary differential equation, updates or accumulates the estimated image detail every time the neural network iterates, and calculates the cumulative result. A training process of adding to the first input image can be performed. In addition, the parameters may be optimized by performing such a training process until a loss function preset based on different scale factor sets becomes a minimum value or a preset reference value or less.

Meanwhile, the above-described providing step (S82) may basically use scale factors that gradually decrease, but are not limited thereto, and scale factors that gradually increase in variations according to the type, shape, content, etc. of the original super-resolution target image. It can be implemented to use the factors sequentially.

9 is a diagram showing a comparison of super-resolution results of the super-resolution method (NODE-RDN) of this embodiment and the super-resolution methods (DRRN, SRFBN) of the comparative example. 10 is a diagram showing a comparison of another super-resolution result of a super-resolution method (NODE-RDN) of this embodiment and super-resolution methods (DRRN, SRFBN) of a comparative example. 11 is a diagram showing a comparison of super-resolution results of the super-resolution method (NODE-RDN) of the present embodiment and the super-resolution method (Meta-RDN) of the comparative example.

Referring to FIG. 9, the method (NODE-RDN) of this embodiment is an initial input by upscaling the original image (GT) for a specific part of a predetermined image (img_012) obtained from a specific database (Urban100) by 2 times (x2) It shows the result of applying multiple scale factors (1,75; 1.5; 1.25; 1) that gradually decrease linearly over 4-stages to the image (see Bicubic image). Multiscale factors are non-integer except for 1.

The super-resolution result by the method of this embodiment, when compared with the final super-resolution result of Comparative Example 1 (DRRN: deep recursive residual network) under the condition of using the same network parameters, the SR performance of this embodiment can be seen simply by eye. It can be seen that the SR performance of Comparative Example 1 is very superior. The final super-resolution result of Comparative Example 1 is the result obtained after repeating 25 times (#it) according to the network settings.

In addition, it can be confirmed that the super-resolution result obtained by the method of this embodiment shows similar performance to each other when compared to the final result of Comparative Example 2 (SRFBN: super-resolution feedback network) under the condition of using the same network parameters. (See Table 1). The final result of Comparative Example 2 is a super-resolution result obtained through repeated feedback 4 times (#it).

Next, referring to FIG. 10, the method (NODE-RDN) of the present embodiment upscals the original image (GT) for a specific part of a predetermined image (img_092) obtained from a specific database (Urban100) by 4 times (x4). Four resulting images, including three intermediate images, are shown by applying multi-scale factors (3; 2; 1.5; 1) that gradually decrease non-linearly over four stages to one initial input image (see Bicubic image). Multiple scale factors include non-integers (1.5).

When comparing the super-resolution result of the method of this embodiment with the super-resolution result of the network in which the number of repeating blocks (#it) is 10, 15, 20, and 25 in Comparative Example 1 (DRRN), simply by eye Even looking at it, it can be seen that the performance of this example is very superior to that of Comparative Example 1.

In addition, when the super-resolution result by the method of this embodiment is compared with the super-resolution results of the network in which the number of feedbacks (#it) is 1, 2, 3, and 4 in Comparative Example 2 (SRFBN), the method of this embodiment It can be seen that the performance by is superior to that of Comparative Example 2.

Comparative Example 3 (DRCN: Deeply-recursive convolutional network) and Comparative Example 4 (LapSRN: Laplacian pyramid super-resolution network) and Comparative Example 5 (D-DBPN: Dense-deep back-projection network) by comparing the network super-resolution (SR) performance of this embodiment (NODE-RDN, NODE-RDN +) SR performance It is shown in Table 1 below.

In Table 1, average PSNR (peak signal-to-noise ratio) and SSIM for low-resolution bicubic images to which three types of scale factors (x2, x3, x4) are applied to the performance of the present embodiment and comparative examples. (structural similarity index) values. Values marked with highlights or shaded backgrounds represent the best performance results and the second best performance results.

As described above, the method of the present embodiment can manage a continuous scale factor in a linear or non-linear aspect during the SR task. That is, as shown in Table 2, it is possible to manage and use linear multi-scale factors set at intervals of 0.1 from 1.1 to 4.

As shown in Table 2, compared to the SR methods (VDSR, VDSR + t, RDN, RDN + t, Meta-RDN) of the comparative example, the SR methods (NODE-VDSR, NODE-RDN, NODE-RDN+ It can be seen that the performance of ) is superior to that of the SR methods of the Comparative Example at every scale.

VDSR+t and RDN+t of the comparative example used for fair comparison are versions of VDSR and RDN modified to have the same input resolution and output resolution by taking the scale factor (t) as an additional input of the network.

In addition, as shown in FIG. 11, when compared with the clean original image GT for a specific part of the specific images 196073 and 48026 of the dataset B100, the original clean image GT is multiplied by 2.5 times (x2. 5) and 4 times (x4) magnification, respectively, looking at the result of super-resolution processing of the original image (Bicubic image) of the super-resolution target, the final image output through the SR method (NODE-RDN) of this embodiment is Meta of the comparative example -You can see that the edges are more clearly recovered than the image output from RDN.

12 is a graph showing the performance (PSNR) evaluated while changing the scale factor in the super-resolution methods (NODE-RDN, NODE-VDSR) of the present embodiment.

12, the scale factor is changed from 1.1 to 4.5 at 0.05 intervals. Also, dot-shaped marks correspond to scale factors seen during the training process (e.g. 1.1, 1.2, …, 4.0), and cross-shaped marks correspond to scale factors seen during the training process. (unseen scale factors) (e.g. 1.15, 1.25, ..., 4.5).

Referring to FIG. 12, as a result of measuring PSNR values in the SR methods of this embodiment while changing the scale factor for a specific image of the B100 dataset, the SR methods of this embodiment learn interpolation ability, It can be seen that unseen scale factors between 1 and 4 (eg, 1.15, 1.25, ..., 3.95) are substantially continuously processed (see the curve for Bicubic in the comparative example).

Moreover, it can be seen that the SR methods of the present embodiment learn extrapolation ability and treat invisible scale factors greater than 4 (eg 4.1, 4.2, ..., 3.95) substantially continuously.

As such, it can be seen that the SR methods of the present embodiment can have power of generalization such as interpolation or extrapolation even if they are trained with only a limited number of scale factors.

In addition, the SR methods of this embodiment have substantially the same SR performance, as shown in Table 3 above, even when different ODE solvers such as the Euler method and the Runge-Kutta method are used. It can be seen that represents However, since the quality of the final output image or the estimated high-resolution image may depend on the performance of the ODE solver employed, it is required to select an appropriate ODE solver.

13 is a graph for explaining a principle of setting a scale factor that can be employed in the image super-resolution method of the present embodiment based on the strength of intermediate image residuals.

Referring to FIG. 13 , in relation to a super-resolution method for a first original image of an eye region and a second original image of a cheek region of a human face, a neural network during an incremental SR process in a test step When the scale factor decreases from 4 to 1 and the initial condition is 4 to check the intermediate result of , calculating the L1 loss, which is the absolute value or the sum of the absolute values of the neural network, the high-frequency patch or Sharp corresponding to the first original image The absolute value of the sharp patch is relatively larger than that of the low-frequency patch when the scale factor is small, and the absolute value of the low-frequency patch or homogeneous patch corresponding to the second original image is large when the scale factor is large. It can be seen that it is considerably larger than that of the high-frequency patch.

In a gradually decreasing scale factor, when the scale factor is small, the output image of the neural network approaches the clean original image, and at this time, the image difference between the input image and the output image of the neural network contains more high-frequency components. Considering that, the absolute value of the neural network for the image of the eye area including high-frequency detail is large when the scale factor is small, and the absolute value of the neural network for the image of the cheek area that does not require high-frequency detail is small when the scale factor is small. It can be seen that when the

For this reason, the scale factor employed in the present embodiment is not limited to a configuration in which rational numbers gradually decrease from the initial scale factor and are finally set to 1. That is, according to a modification of the present embodiment, the scale factor is a scale factor corresponding to the size of the first input image of the low-resolution super-resolution target, that is, a scale of a rational number that gradually increases from a value of 1 to a scale factor of 1 at preset intervals. factors, and a final scale factor corresponding to the size of the relative high resolution, which is the desired resolution after the super-resolution process. Here, the value of the first scale factor of the gradually decreasing scale factor may be the same as the value of the final scale factor of the gradually increasing scale factor.

In addition, considering the super-resolution process according to the setting of the scale factor of the present embodiment, the gradual image resolution improvement method or the image super-resolution method of the present invention converts the original super-resolution target image into a first part including a plurality of high-frequency areas and a low-frequency area. After improving the resolution with sets of different scale factors for the first original image and the second original image from which the second part including a plurality of isolates are separated, that is, after performing the super-resolution operation, the super-resolution operation is completed. A process of combining the first partial image and the second partial image and, if necessary, softening the boundary between the first partial image and the second partial image through a filter such as a Gaussian smoothing filter, a low-frequency filter, and a deblocking filter may be added. can

The operation of the method according to the embodiment of the present invention can be implemented as a computer readable program or code on a computer readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. In addition, computer-readable recording media may be distributed to computer systems connected through a network to store and execute computer-readable programs or codes in a distributed manner.

In addition, the computer-readable recording medium may include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. The program command may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine code generated by a compiler.

Although some aspects of the present invention have been described in the context of an apparatus, it may also represent a description according to a corresponding method, where a block or apparatus corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method may also be represented by a corresponding block or item or a corresponding feature of a device. Some or all of the method steps may be performed by (or using) a hardware device such as, for example, a microprocessor, programmable computer or electronic circuitry. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

In embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In embodiments, a field programmable gate array may operate in conjunction with a microprocessor to perform one of the methods described herein. Generally, methods are preferably performed by some hardware device.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention described in the claims below. You will understand that you can.

Claims (15)

Starting from the 1st input image obtained by upscaling the original input image, the N-1th (N is a natural number greater than or equal to 2) input image is input to the neural network, and a scale factor is applied to obtain the N-1th input image in the neural network. Outputting a residual image (a);
(b) generating an N-th input image by adding the N-1-th residual image and the N-1-th input image, wherein the N-th input image is transmitted as an input of the neural network; and
Step (c) of repeating steps (a) and (b) while gradually reducing the size of the scale factor by a preset size until the scale factor becomes 1;
When the scale factor is applied as 1, a final output image is generated as a sum of the first input image and the cumulative ordinary differential of the image residuals between the N-1th residual image and the Nth residual image. How to improve image resolution.
The method of claim 1,
The scale factor is a rational number value greater than 1, the gradual image resolution enhancement method.
The method of claim 2,
The scale factor is an input added to the neural network, and is a value that sequentially decreases in a geometrical, linear, geometrical, or nonlinear manner for each repetitive operation of the neural network.
The method of claim 1,
The cumulatively calculated value is modeled as an ordinary differential equation (ODE) represented by Equation 4 below,
[Equation 4]

In Equation 4,

is the final output image,

Is the first input image to which the initial scale factor (t _o ) is applied,

is an image to which a gradually reduced scale factor (t) is applied to the input image of the neural network, and f()dt is an image residual between the input image and the output image of the neural network to which the scale factor (t) is additionally input. an ordinary differential equation,

Wherein denotes a trainable parameter of the neural network, progressive image resolution enhancement method.
The method of claim 4,
The cumulatively calculated value is calculated numerically by an ODE solver,
The ODE solver includes one or more selected from the Euler method, the Runge-Kutta method, the Heun method, the midpoint method, and the adaptive Runge-Kutta method.
The method of claim 4,
The neural network trains the parameters to minimize the loss function represented by Equation 6,
[Equation 6]

In Equation 6,

is the parameter

The equation for minimizing the loss function by training

A gradual image resolution enhancement method in which HR represents an original image for training and ODESolove() represents an ODE solver function, respectively.
The method of claim 6,
The step of updating parameters of the neural network through backpropagation, the progressive image resolution enhancement method.
As a gradual image resolution enhancement device,
processor;
a memory electronically coupled to the processor; and
contains instructions stored in the memory;
When executed by the processor, the instructions cause the processor to:
Starting with the 1st input image obtained by upscaling the original input image, input the N-1th (N is a natural number greater than or equal to 2) input image to the neural network and apply a scale factor to the N-1th input image in the neural network. Outputting a residual image (a);
(b) generating an N-th input image by adding the N-1-th residual image and the N-1-th input image, wherein the N-th input image is transmitted as an input of the neural network;
(c) repeatedly performing steps (a) and (b) while gradually decreasing the size of the scale factor by a preset size until the scale factor becomes 1; and
generating a total output image as a sum of a value obtained by cumulatively calculating image residuals between the N-1 th residual image and the N th input image when the scale factor is applied as 1 and the first input image (d) A gradual image resolution enhancement device for performing
The method of claim 8,
The cumulatively calculated value is modeled as an ordinary differential equation (ODE),
[Equation 4]

In Equation 4,

is the final output image,

Is the first input image to which the initial scale factor (t _o ) is applied,

is an image to which a gradually reduced scale factor (t) is applied to the input image of the neural network, and f()dt is an image residual between the input image and the output image of the neural network to which the scale factor (t) is additionally input. an ordinary differential equation,

represents a trainable parameter of the neural network, a gradual image resolution enhancement device
The method of claim 9,
The neural network trains the parameters to minimize the loss function represented by Equation 6,
[Equation 6]

In Equation 6,

is the parameter

The equation for minimizing the loss function by training

is a relatively high resolution (HR) training original image, and ODESolove() represents an ODE solver function, respectively, a progressive image resolution enhancement device
In the image super-resolution method performed by the processor,
re-inputting an output image of a neural network into which an input image is input as an input image of the neural network;
sequentially providing scale factors, the sizes of which are gradually set to be different, to the neural network whenever the input image is input to the neural network;
calculating an image residual between an input image and an output image of the neural network by using an ordinary differential equation for each repetitive operation of the neural network and accumulating or updating the calculated values; and
generating a final output image by adding the final image residual obtained in the updating step and the initial input image;
Image super-resolution method comprising a.
The method of claim 11,
The providing step further inputs the scale factor to the first layer of the neural network in addition to the input image.
The method of claim 12,
In the inputting and providing, padding is added to an edge of the input image in correspondence to a scale factor having a size of a rational number greater than 1 based on the input image, and the padding is added to correspond to the input image. An image super-resolution method that center-crops the input image.
The method of claim 11,
The neural network includes a residual dense network (RDN), and the RDN has a form in which a last upsampling layer is omitted so that an input resolution and an output resolution are identical.
The method of claim 11,
Further comprising training the neural network,
The training may include estimating image details through a difference in image residuals between an input image and an output image of the neural network through the ordinary differential equation, updating the image detail each time the neural network is executed, and adding the image to the first input image. An image super-resolution method in which a single training process is performed for different sets of scale factors to train a preset loss function to be a minimum value or less than a preset reference value.

Images