KR20210116922A - Method and Device for Fast Adaptation through Meta-learning of Super Resolution Model - Google Patents

Abstract

Disclosed is a method and an apparatus for fast adaptation through meta-learning of a super-resolution model. According to an aspect of the present invention, a method for learning a neural network implemented by a computer system comprises the steps of: generating a low-resolution image and a medium-resolution image for each of a plurality of high-resolution sample images; generating a plurality of task networks corresponding to each low-resolution image by training the neural network to receive the low-resolution image and output a corresponding medium-resolution image, with respect to each low-resolution image; calculating a loss of the task network which receives the medium-resolution image and outputs a corresponding high-resolution sample image, with respect to each medium-resolution image; and generating a meta-learned neural network by updating parameters of the neural network based on the losses calculated from the plurality of task networks. The present invention can efficiently generate the high-resolution image.

Description

Method and Device for Fast Adaptation through Meta-learning of Super Resolution Model

Embodiments of the present invention relate to a super-resolution model in which a neural network performs meta-learning to improve the resolution of several images in a learning stage, and performs a fast adaptation process in a test stage.

The content described in this section merely provides background information on the present invention and does not constitute the prior art.

The super-resolution model refers to a neural network that generates a high-resolution image by improving the resolution of a low-resolution image. The super-resolution model is used to process image or video data in various electronic devices such as smartphones, TVs, and PCs.

In general, a technique for improving the resolution of a low resolution image to a high resolution is a technique using an interpolation method. The interpolation method is a method of filling pixels of a high-resolution image from pixels of a low-resolution image through a 'nearest neighbor' method, a 'bi-cubic' method, or the like. However, when the interpolation method is used, there is a problem that the output image has a high resolution but looks blurry to the naked eye.

In order to solve this problem, an image of various focus may be taken using dedicated hardware that assists the camera, and a clear high-resolution image may be output from several images, but additional costs are incurred due to the hardware.

In order to compensate for the shortcomings of the interpolation method and to improve the performance of the super-resolution model without additional hardware, deep learning-based single image super-resolution (SISR) research is being actively conducted. Here, single-image super-resolution is a technique for increasing the resolution of a single image by using only features included in the single image. In particular, when SISR uses a convolutional neural network based on deep learning, the performance of SISR can be improved without adding hardware.

However, the existing deep learning-based super-resolution model is trained using a vast training data set in the learning stage, and in the test stage, it does not effectively improve the resolution of the test image because it receives a test image that is not used in the learning stage. This is because most SISRs using deep learning do not adjust the parameters of the super-resolution model during the test phase. In other words, the use of fixed network parameters for all test images without further learning the features included in the new test image is a factor that degrades the performance of the super-resolution model.

Therefore, it is necessary to learn about various images in the training stage, but to generate high-resolution images by using the features included in each image in the test stage.

On the other hand, in the existing super-resolution restoration method that does not use a neural network, a low-resolution image is given to a user, and the user directly intervenes to generate a high-resolution image from the low-resolution image. In this case, if there is a region that is repeated with different sizes in a given low-resolution image, such as a photograph of a natural landscape or a building, this region is used to generate a high-resolution image. For example, in a photograph of a building, when windows of the same shape and different sizes exist, the high-resolution image of the small window is assumed to be the large window. That is, a large window in a low-resolution image becomes a small window in a high-resolution image. Users can efficiently generate high-resolution images using such patch recurrence.

However, a user can create a high-resolution image, but there is a limit in increasing the resolution of the high-resolution image because not all patches in a given image are similar to each other.

Embodiments of the present invention provide a neural network learning method and apparatus for performing meta-learning in the learning stage of the neural network for fast adaptation in the test stage without changing the architecture of the existing super-resolution model Its main purpose is to provide

Other embodiments of the present invention provide a method and apparatus for using a neural network using dynamic parameters by improving the resolution of a test image after adapting a meta-learned neural network to a given test image in the test step in the learning step. The purpose is to provide

Another embodiment of the present invention aims to provide a neural network learning method and apparatus in which a user indirectly learns a patch recall region without direct intervention by learning the neural network on the entire image.

According to an aspect of the present invention, there is provided a method for learning a neural network implemented by a computer system, comprising: generating a low-resolution image and a medium-resolution image for each of a plurality of high-resolution sample images; generating a plurality of task networks corresponding to each low-resolution image by training a neural network to receive a low-resolution image and output a corresponding medium-resolution image for each low-resolution image; for each medium-resolution image, calculating a loss of a task network that receives a medium-resolution image and outputs a corresponding high-resolution sample image; and generating a meta-learned neural network by updating parameters of the neural network based on losses calculated from a plurality of task networks.

According to another aspect of this embodiment, there is provided a method for generating a high-resolution image based on a neural network implemented by a computer, the method comprising: acquiring an input image; generating a low-resolution input image for the input image; adapting the meta-learned neural network to receive the low-resolution input image and output the input image; obtaining an output image with improved resolution by inputting the input image to a neural network that has been adapted; and outputting the output image.

As described above, according to an embodiment of the present invention, by performing meta-learning in the learning stage of the neural network previously learned for various images, the performance of the neural network for various images is also improved without changing the architecture, and the test stage is performed. can increase the speed of adaptation in

According to another embodiment of the present invention, by generating a high-resolution image after performing an adaptation process on a given test image in the test stage of the meta-learning neural network for various images, it is adaptively clear to individual test images. It can create high-resolution images.

According to another embodiment of the present invention, a high-resolution image can be efficiently generated by using a neural network that learns a relationship between patches similar to each other for a low-resolution image including patches similar to each other.

1 is a diagram illustrating a patch recall according to an embodiment of the present invention.
2 is a diagram illustrating a process of performing a fast adaptation method through meta-learning according to an embodiment of the present invention.
3 is a diagram illustrating a meta-learning training method of a neural network according to an embodiment of the present invention.
4 is a diagram illustrating an adaptation process in a test step of a meta-learned neural network according to an embodiment of the present invention.
5 is a flowchart illustrating an operation process of an apparatus for learning a neural network according to an embodiment of the present invention.
6 is a flowchart illustrating an operation process of an apparatus for generating a high-resolution image based on a neural network according to an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

In addition, in describing the components of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the components from other components, and the essence, order, or order of the components are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. . In addition, terms such as '~ unit' and 'module' described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

1 is a diagram exemplified to explain a conventional patch recall.

Patch-recurrence refers to a method of using one pattern in an image when creating a high-resolution image for an image in which patterns similar to each other are repeated. For example, if some areas are repeated in a low-resolution image, such as a natural image with aligned street trees or a building image with aligned windows, using the repeated area can be efficiently used to create images with different resolutions. can create However, in the method of using the existing patch reference, the user must directly select and adjust similar patches.

Referring to FIG. 1 , a low-resolution image 100 , a medium-resolution image 110 , a high-resolution image 120 , a low-resolution patch 104 , a first patch 112 , a second patch 114 , and a high-resolution patch 122 ) is shown. The low-resolution image 100 , the medium-resolution image 110 , and the high-resolution image 120 correspond to each other and constitute one training data set.

A user determines a first patch 112 that is a partial region within a given medium-resolution image 110 . The first patch 112 may be a region including one of the repeating patterns in the medium-resolution image 110 . Thereafter, the learning apparatus detects the low-resolution patch 104 , which is a region having a high similarity to the first patch 112 in the low-resolution image 100 .

The user detects the second patch 114 corresponding to the low-resolution patch 104 in the medium-resolution image 110 . The second patch 114 is the region used to create the low resolution patch 104 .

A user creates a high-resolution patch 122 at a position corresponding to the first patch 112 by using the second patch 114 .

Although the user can create a high-resolution image through the above-described process, there is a limit in increasing the resolution of the high-resolution image because not all patches in a given image are similar to each other.

2 is a diagram illustrating a process of performing a fast adaptation method through meta-learning according to an embodiment of the present invention.

Referring to FIG. 2 , the rapid adaptation method through meta-learning is divided into a training phase and a test phase, and the training phase includes (a) network initialization and (b) meta-learning ( It is divided into meta-learning stage, and the test stage is divided into (c) adaptation stage and (d) resizing stage. In the training stage, a learning apparatus (not shown) of the neural network may be used, and in the testing stage, an image generating apparatus (not shown) of the neural network may be used. For example, the learning apparatus may be implemented as a server and the image generating apparatus may be implemented as a terminal. However, this is only an example, and the two devices may be implemented as one device. Meanwhile, the network initialization step may be performed in advance by an external device other than the learning device.

(a) The network initialization step is a step of pre-training the neural network 200 to generate high-resolution images for a large amount of image data before performing meta-learning of the neural network 200 . Here, the neural network 200 is a component that receives a low-resolution image and outputs a high-resolution image. In the learning process, parameters of the neural network 200 are initialized.

(b) The meta-learning step is a step in which the initial neural network 210 learns images that are distinguished into various tasks. The initial neural network 210 receives a high-resolution sample image (H) for each of a plurality of high-resolution sample images, generates a low-resolution image (L) and a medium-resolution image (M), and then learns the three images. By repeating this process for a plurality of high-resolution sample images, meta-learning is performed.

Meanwhile, in the meta-learning stage, the learning device generates a low-resolution image and a medium-resolution image from a high-resolution sample image, and there is no need to receive a low-resolution image from an external source. Also, unlike a vast amount of images used in the network initialization step, images including patch references such as natural landscapes or building images can be efficiently learned. In addition, since the learning apparatus performs meta-learning in the training phase, the image generating apparatus may further improve the network adaptation speed in the adaptation phase.

(c) The adaptation step is a step in which the image generating device fine-tunes parameters of the network by adapting the meta-learned neural network 220 to input images for each image generating device. Specifically, the image generating apparatus acquires the input image I and generates a down-sampling low-resolution input image. The meta-learned neural network 220 is adapted to the input image by repeatedly learning the low-resolution input image and the input image a preset number of times. In the adaptation step, the adaptation speed is accelerated by the aforementioned meta-learning step performed immediately before.

(d) The resizing step is a step of generating an output image O in which the resolution of the input image I is increased using the finally adapted neural network 230 . The adapted neural network 230 generates and outputs a high-resolution image for the input image used in the adaptation step.

Through the above-described process, a high-resolution, clear image may be generated for a small number of images owned by each image generating device.

3 is a diagram illustrating a meta-learning training method of a neural network according to an embodiment of the present invention.

Referring to FIG. 3 , a plurality of high-resolution sample images 300 , a plurality of medium-resolution images 302 , a plurality of low-resolution images 304 , an initial neural network 310 , a plurality of task networks 320 , and meta-learned A neural network 330 is shown.

As in the network initialization step in FIG. 2 , the initial neural network 310 may be a neural network pre-trained to generate an output image with improved resolution for an arbitrary input image. Specifically, the initial neural network is trained so that the loss function between the output image and the correct answer image generated for the input image becomes small.

Referring back to FIG. 3 , the plurality of high-resolution sample images 300 are given images, and the plurality of medium-resolution images 302 and the plurality of low-resolution images 304 are generated to correspond to the plurality of high-resolution sample images 300 . are images. Specifically, the low-resolution image and the medium-resolution image are generated by at least one of a nearest neighbor method, a bi-linear method, or a bi-cubic method from a high-resolution sample image by another neural network. It may be generated using a down-sampling method.

The low-resolution images L1, L2, L3, and L4 and the plurality of medium-resolution images M1, M2, M3, and M4 correspond to the high-resolution sample images H1, H2, H3, and H4, respectively. That is, a high-resolution sample image, a medium-resolution image, and a low-resolution image constitute one training data pair, and are used for meta-learning of the initial neural network 310 . The number of training data pairs corresponds to the number of task networks used for meta-learning.

)를 갱신함으로써 각 저해상도 이미지에 대응되는 복수의 태스크 네트워크(320)를 생성한다. For each low-resolution image, the learning device receives a low-resolution image as an input and trains the initial neural network 310 to output a medium-resolution image corresponding to the low-resolution image, that is, the parameters of the initial neural network 310 (

) to generate a plurality of task networks 320 corresponding to each low-resolution image.

)에 반영하여, 제1 태스크 파라미터(

)를 갖는 제1 태스크 네트워크(322)를 생성한다. 즉, 제1 태스크 네트워크(322)는 초기 뉴럴 네트워크(310)가 L1 이미지 및 M1 이미지에 대해 학습한 결과이고, 제2 태스크 네트워크(324)는 L2 이미지 및 M2 이미지에 대해 학습한 결과다. 이를 내부 루프라고 지칭하며, 수학식으로 나타내면 수학식 1과 같다.Specifically, the learning apparatus inputs one low-resolution image L1 to the initial neural network 310 and acquires a medium-resolution output image generated by the initial neural network 310 . The learning apparatus calculates a loss based on a difference between the medium-resolution output image and the medium-resolution image M1. The learning device uses the calculated loss as a parameter (

), reflected in the first task parameter (

) to create a first task network 322 with That is, the first task network 322 is the result of the initial neural network 310 learning the L1 image and the M1 image, and the second task network 324 is the learning result of the L2 image and the M2 image. This is referred to as an inner loop, and is expressed as Equation (1).

는 i번째 태스크 네트워크의 파라미터,

는 초기 뉴럴 네트워크의 파라미터,

는 학습률(learning rate),

는 그래디언트(gradient), L은 손실함수,

는 저해상도 이미지,

는 중해상도 출력 이미지를 의미한다.in Equation 1

is the parameter of the i-th task network,

is the parameters of the initial neural network,

is the learning rate,

is the gradient, L is the loss function,

is a low-resolution image,

denotes a medium-resolution output image.

Then, the learning apparatus calculates a plurality of losses from the plurality of task networks 320 . Specifically, for each medium-resolution image, the learning apparatus calculates a loss of a task network that receives a medium-resolution image and outputs a high-resolution sample image corresponding to the medium-resolution image. For example, when the first task network 322 receives the medium-resolution image M1 as an input, it generates a high-resolution output image. In this case, the learning apparatus calculates a loss based on the difference between the high-resolution output image and the high-resolution image H1.

The learning apparatus generates a meta-learned neural network 330 by updating the parameters of the initial neural network 310 based on the plurality of losses calculated from the plurality of task networks 320 . That is, the meta-learning neural network 330 is a result of reflecting the learning result of the plurality of task networks 320 in the initial neural network 310 . This is referred to as an outer loop, and is expressed as Equation (2).

는 메타 러닝된 뉴럴 네트워크의 파라미터,

는 학습률,

는 i번째 태스크 네트워크에 의해 생성된 고해상도 출력 이미지,

는 고해상도 샘플 이미지다.in Equation 2

is the parameter of the meta-learned neural network,

is the learning rate,

is the high-resolution output image generated by the i-th task network,

is a high-resolution sample image.

The neural network 330 meta-learned through the above-described process has superior performance in generating a high-resolution image for the input image compared to the initial neural network 310 . In addition, the speed at which the meta-learned neural network 330 adapts to the test image in the test phase is much faster than that of the initial neural network 310 .

4 is a diagram illustrating an adaptation process and an inference process in a test step of a meta-learned neural network according to an embodiment of the present invention.

Referring to FIG. 4 , the meta-learned neural network 330 , the input image 400 , the low-resolution input image 410 , the medium-resolution output image 420 , the adapted neural network 430 and the output image 440 are is shown.

The neural network-based image generating apparatus generates a down-sampled low-resolution input image 410 from the acquired input image 400 in order to adapt the meta-learned neural network 330 to the input image 400 . do. In this case, the low-resolution input image 410 is down-sampled from the input image 400 by at least one of a nearest neighbor method, a bi-linear method, or a bi-cubic method. -sampling) may be an image.

The image generating apparatus adapts the meta-learned neural network 330 to receive the low-resolution input image 410 and generate the input image 400 . Specifically, the image generating apparatus obtains the medium-resolution output image 410 by inputting the low-resolution input image 410 into the meta-learning neural network 330 . Then, the image generating apparatus calculates a loss function based on the difference between the medium-resolution output image 420 and the input image 400, and updates the parameters of the meta-learned neural network 330 based on the calculated loss function ( adjust). This process may be repeatedly performed until a preset condition such as a preset number of times or a preset loss function is satisfied. This adaptation process is accelerated by meta-learning performed immediately before.

When the adaptation process for the input image 400 is completed, the image generating apparatus inputs the input image 400 to the adapted neural network 430 and obtains an output image 440 with improved resolution.

5 is a flowchart illustrating an operation process of an apparatus for learning a neural network according to an embodiment of the present invention.

The learning apparatus generates a low-resolution image and a medium-resolution image for each of the plurality of high-resolution sample images (S500).

For each of the plurality of low-resolution images, the learning apparatus generates a plurality of task networks by receiving the low-resolution image and updating the parameters of the initial neural network to output the corresponding medium-resolution image (S502). In this case, the plurality of task networks correspond to each of the plurality of high-resolution sample images. Meanwhile, the initial neural network may be trained in advance to generate an output image with improved resolution from an arbitrary input image.

The learning apparatus receives the medium-resolution image and calculates the losses of the task network that outputs the corresponding high-resolution sample image (S504). The task network loss is calculated based on the difference between the high-resolution output image output by the task network receiving the medium-resolution image and the high-resolution sample image.

The learning apparatus generates a meta-learned neural network by updating the parameters of the initial neural network based on the losses calculated from the plurality of task networks (S506).

6 is a flowchart illustrating an operation process of an apparatus for generating an image of a neural network according to an embodiment of the present invention.

The image generating apparatus acquires an input image and generates a low-resolution input image with respect to the input image (S600). The meta-learned neural network can be adapted to both the input image and the low-resolution input image, so that it can perform optimally when generating the final output image.

The image generating apparatus performs an adaptation process to output an input image by receiving a low-resolution input image for the meta-learning neural network ( S602 ). The adaptation process is a process of calculating a loss based on the difference between the medium-resolution output image and the input image output by the meta-learning neural network, and then updating the parameters of the meta-learned neural network based on the loss. This may be repeatedly performed until a preset condition is satisfied.

The image generating apparatus inputs an input image to the neural network for which adaptation is completed, obtains an output image with improved resolution, and then outputs it ( S604 ).

Although it is described that steps S500 to S604 are sequentially executed in FIGS. 5 and 6 , this is merely illustrative of the technical idea of an embodiment of the present invention. In other words, those of ordinary skill in the art to which an embodiment of the present invention pertain change the order described in FIGS. Since it will be possible to apply various modifications and variations by executing one or more processes in parallel in S604, FIGS. 5 and 6 are not limited to a time-series order.

Meanwhile, the processes shown in FIGS. 5 and 6 can be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data readable by a computer system is stored. That is, such a computer-readable recording medium may be a non-transitory medium such as ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and also carrier wave (for example, , transmission over the Internet) and may further include a transitory medium such as a data transmission medium. In addition, the computer-readable recording medium is distributed in a network-connected computer system so that the computer-readable code can be stored and executed in a distributed manner.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

100: neural network 210: initial neural network
220: multiple task network 230: meta-learned neural network
330: adapted neural network

Claims (11)

A method for learning a neural network implemented by a computer system, comprising:
generating a low-resolution image and a medium-resolution image for each of the plurality of high-resolution sample images;
generating a plurality of task networks corresponding to each low-resolution image by training a neural network to receive a low-resolution image and output a corresponding medium-resolution image for each low-resolution image;
for each medium-resolution image, calculating a loss of a task network that receives a medium-resolution image and outputs a corresponding high-resolution sample image; and
generating a meta-learned neural network by updating parameters of the neural network based on losses calculated from a plurality of task networks;
learning methods that include According to claim 1,
wherein the neural network is pre-trained to generate an output image with improved resolution with respect to an input image. According to claim 1,
The process of creating the plurality of task networks includes:
for each low-resolution image, inputting the low-resolution image into the neural network to obtain a medium-resolution output image;
calculating a loss based on a difference between the medium-resolution output image and a corresponding low-resolution image; and
generating the plurality of task networks by updating parameters of the neural network for each loss corresponding to each low-resolution image;
learning methods that include According to claim 1,
The process of calculating the loss of the task network is
obtaining a high-resolution output image by inputting a medium-resolution image corresponding to the task network into the task network; and
calculating a loss based on a difference between the high-resolution output image and a corresponding medium-resolution image;
learning methods that include According to claim 1,
The low-resolution image and the medium-resolution image are,
A learning method that is an image down-sampled from the high-resolution sample image by at least one of a nearest neighbor method, a bilinear method, or a bicubic method. In the neural network-based high-resolution image generation method implemented by a computer,
acquiring an input image;
generating a low-resolution input image for the input image;
adapting the meta-learned neural network to receive the low-resolution input image and output the input image;
obtaining an output image with improved resolution by inputting the input image to a neural network that has been adapted; and
outputting the output image;
A method for generating high-resolution images, including 7. The method of claim 6,
The meta-learned neural network is
generating a low-resolution image and a medium-resolution image for each of the plurality of high-resolution sample images;
generating a plurality of task networks corresponding to each low-resolution image by training a neural network to receive a low-resolution image and output a corresponding medium-resolution image for each low-resolution image;
for each medium-resolution image, calculating a loss of a task network that receives a medium-resolution image and outputs a corresponding high-resolution sample image; and
generating a meta-learned neural network by updating parameters of the neural network based on losses calculated from a plurality of task networks;
A method of generating high-resolution images that is learned including In the neural network learning apparatus,
at least one memory for storing instructions; and
at least one processor for processing audio data by executing at least one instruction stored in the memory;
The at least one memory through the at least one processor,
generating, for each of the plurality of high-resolution sample images, a low-resolution image and a medium-resolution image;
For each low-resolution image, by training a neural network to receive a low-resolution image and output a corresponding medium-resolution image, a plurality of task networks corresponding to each low-resolution image are generated, and the neural network has an improved resolution for the input image. pre-trained to generate output images;
For each medium-resolution image, calculating the loss of the task network that receives the medium-resolution image and outputs a corresponding high-resolution sample image, and
A learning apparatus configured to generate a meta-learned neural network by updating parameters of the neural network based on losses calculated from a plurality of task networks. In the neural network-based high-resolution image generating apparatus,
at least one memory storing instructions; and
at least one processor for processing audio data by executing at least one instruction stored in the memory;
The at least one memory through the at least one processor,
acquiring an input image;
generating a low-resolution input image for the input image;
adapting the meta-learned neural network to receive the low-resolution input image and output the input image;
obtaining an output image with improved resolution by inputting the input image to a neural network that has been adapted; and
outputting the output image;
A high-resolution image-generating device set to perform 10. The method of claim 9,
The learning method of the meta-learned neural network is,
generating a low-resolution image and a medium-resolution image for each of the plurality of high-resolution sample images;
generating a plurality of task networks corresponding to each low-resolution image by training a neural network to receive a low-resolution image and output a corresponding medium-resolution image for each low-resolution image;
for each medium-resolution image, calculating a loss of a task network that receives a medium-resolution image and outputs a corresponding high-resolution sample image; and
generating a meta-learned neural network by updating parameters of the neural network based on losses calculated from a plurality of task networks;
A high-resolution image generating device comprising a. A computer-readable recording medium recording a program for executing the method of any one of claims 1 to 7 on a computer.

Images