KR102299956B1 - Image quality improvement apparatus using artificial neural network - Google Patents

Abstract

The image quality improving apparatus includes a model generator generating an adversarial generated neural network model including a generator generating a similar high-definition sample from a first image and a discriminator determining whether the similar high-definition sample is authentic or not; A learning unit that extracts a sample, downscales the sample, and improves the image quality of the downscaled sample to generate a pseudo-high-definition sample, and trains the adversarial generated neural network model so that the discriminator determines whether the pseudo-high-definition sample is authentic or not. and an image quality improvement unit that improves the quality of the second image by inputting the second image to the adversarial generative neural network model.

Description

Image quality improvement device using artificial neural network {IMAGE QUALITY IMPROVEMENT APPARATUS USING ARTIFICIAL NEURAL NETWORK}

The present invention relates to an apparatus for improving image quality using an artificial neural network.

Deep learning technology is widely used in various fields such as computer vision, speech recognition, and image quality improvement.

The recently proposed ZSSR (Zero-Shot Super-Resolution) model as a method of image quality improvement creates a new high-quality image through internal learning from a single high-quality image. However, in the case of the ZSSR model, it is difficult to obtain a generalized image quality improvement model because it is learned from a single image, so there is a possibility of learning a distorted model depending on the type of image. In addition, the ZSSR model has a problem in that the performance is somewhat different depending on the type of high-definition image to be learned.

On the other hand, the SRGAN (Super-Resolution GAN) model uses an adversarial generative neural network to improve single image quality. However, since the SRGAN model is not an internal learning structure, a large amount of high-quality images is required to achieve good performance.

Korean Patent Publication No. 2018-0096816 (published on August 29, 2018)

The present invention is to solve the problems of the prior art described above, in which an image quality improvement method using a first image is trained in an adversarial generating neural network model, and the image quality is improved by inputting a second image to the learned adversarial generating neural network model. I would like to provide a second image. However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, an apparatus for improving image quality according to an aspect of the present invention includes a generator for generating a similar high-definition sample from a first image and a discriminator for determining whether the similar high-definition sample is authentic a model generator for generating an adversarial generative neural network model; the generator extracts samples of at least a portion of the first image, downscales the samples, and improves image quality of the downscaled samples to generate the pseudo-high-definition sample, and the discriminator determines the authenticity of the pseudo-high-definition sample. a learning unit that trains the adversarial generative neural network model to determine whether or not; and an image quality improvement unit configured to improve the quality of the second image by inputting a second image to the learned adversarial generative neural network model.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, the present invention trains an adversarial generative neural network model to learn an image quality improvement method using a first image, and inputs the second image to the learned adversarial generative neural network model to improve the image quality. An improved second image may be provided.

Through this, the present invention can improve the performance of image quality improvement by learning the adversarial generative neural network model, and improve the robustness of the adversarial generative neural network model by compensating for the drawbacks of zero-shot image quality improvement.

1 is a block diagram of an apparatus for improving image quality according to an embodiment of the present invention.
2 is a diagram illustrating the structure of an adversarial generative neural network model according to an embodiment of the present invention.
3A to 3C are diagrams for explaining a method of training an adversarial generative neural network model according to an embodiment of the present invention.
4 is a diagram for explaining a method of improving image quality through an adversarial generative neural network model according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise stated.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in the present specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, detailed contents for carrying out the present invention will be described with reference to the accompanying configuration diagram or process flow diagram.

1 is a block diagram of an image quality improvement apparatus 10 using an artificial neural network, according to an embodiment of the present invention.

Referring to FIG. 1 , the image quality improving apparatus 10 may include a model generating unit 100 , a learning unit 110 , and an image quality improving unit 120 . However, the image quality improving apparatus 10 shown in FIG. 1 is only one embodiment of the present invention, and various modifications are possible based on the components shown in FIG. 1 .

Hereinafter, FIG. 1 will be described with reference to FIGS. 2 to 4 together.

The model generator 100 may generate an adversarial generative neural network model for improving image quality based on a Generative Adversarial Network (GAN). Here, the adversarial generative network is a network in which competitive learning is applied to a deep neural network.

Referring to FIG. 2 , the adversarial generation neural network model 200 learns to generate a similar high-definition sample using a high-quality sample in a patch unit randomly extracted from a first high-quality image, and determines whether the generated similar high-definition sample is authentic. A model that learns to discriminate.

The adversarial generative neural network model 200 includes a generator 210 that generates a similar high-definition sample from the first image and a discriminator 220 that determines whether the similar high-definition sample is authentic or not, the generator 210 and the discriminator (220) Each influences the other while learning.

Looking at the structure of the constructor 210, the constructor 210 is formed in a structure in which a fully connected convolutional layer having 64 channels, a batch normalization layer, and a Rectified Linear Unit (RELU) layer are stacked eight times. At this time, a depth convolution layer is used in the structure of the generator 210 in order to reduce the total number of parameters of the adversarial generative neural network structure. Here, since the depth convolution layer has a feature of combining the result values of each channel into one, it is possible to reduce the number of parameters while maintaining the characteristics of each layer.

Looking at the structure of the discriminator 220, the discriminator 220 is formed in a structure in which a fully connected convolutional layer, a batch normalization layer, and a leaky RELU layer are stacked 5 times.

The learning unit 110 may train the generator 210 and the discriminator 220 of the adversarial generative neural network model by using the first image. In addition, the learning unit 110 may train the adversarial generation neural network model so that the generator 210 and the discriminator 220 learn in a zero-shot method.

Specifically, the generator 210 may learn to generate an image similar to the first image based on the adversarial generative network. The generator 210 may learn to generate a similar high-definition sample similar to the high-definition sample extracted from the first image.

The discriminator 220 may learn to distinguish an image similar to the actual first image based on the adversarial generative network. The discriminator 220 may learn to distinguish a similar high-definition sample from a high-quality sample of the actual first image.

3A to 3C, when the first image 301 is input to the adversarial generative neural network model, the learning unit 110 extracts a high-quality sample 303 from the first image 301 by the generator 210, An adversarial generative neural network model can be trained to downscale the extracted high-quality samples 303 . Also, the learning unit 110 may train the adversarial generation neural network model to generate a similar high-quality sample 307 by improving the image quality of the downscaled sample 305 . Here, the similar high-quality sample 307 may be a sample image similar to the high-quality sample 303 extracted from the first image 301 .

The learner 110 performs the generator 210 based on the distance comparison between the feature vector for the high-definition sample 303 extracted from the first image and the feature vector for the similar high-definition sample 307 generated by the generator 210 . The adversarial generative neural network model may be trained while applying the pre-trained high-definition ImageNet weights to the adversarial generative neural network model to generate a sample similar to the high-quality sample 303 of the first image 301 .

The learner 110 may train the generator 210 to minimize the distance between the feature vector of the high-definition sample 303 and the feature vector of the downscaled sample 305 .

The learning unit 110 trains the generator 210 so that the similar high-definition sample 307 generated by the down-scaled sample 305 deceives the discriminator 220 so that the generator 210 is a high-quality sample significantly similar to the real thing. Constructor 210 can be trained to generate 305 .

The learning unit 110 may train the adversarial generated neural network model to determine whether the discriminator 220 is authentic or not of the generated similar high-definition sample 307 .

The learning unit 110 determines whether the similar high-definition sample 307 generated by the discriminator 220 corresponds to the high-quality sample 303 extracted from the actual first image 301 or corresponds to the generated similar-high-definition sample 307 . An adversarial generative neural network model can be trained to determine whether

The generator 210 may learn to minimize the distance between the feature vector of the high-quality sample and the feature vector of the down-scaled sample based on a preset loss function. Here, the preset loss function may be expressed as in [Equation 1], and the preset loss function is a first loss function for adversarial learning between the generator 210 and the discriminator 220 and a previously learned network (eg, VGG network) can be composed of a second loss function representing the distance between the extracted feature vectors. Here, the first loss function may be expressed as [Equation 2], and the second loss function may be expressed as [Equation 3].

[Equation 1]

는 기설정된 손실함수이고,

은 제 1 손실함수이고,

는 제 2 손실함수이고, α는 하이퍼 파라미터이다. here,

is a preset loss function,

is the first loss function,

is the second loss function, and α is the hyperparameter.

[Equation 2]

은 고화질의 제 1 이미지이고, P(

)는 제 1 이미지로부터 추출된 고화질 샘플이고,

은 추출된 고화질 샘플을 다운 스케일링한 샘플이고, G(

)는 다운 스케이링한 샘플의 화질이 개선된 유사 고화질 샘플이고, D(G(

))는 판별자(220)가 유사 고화질 샘플의 진위 여부를 판별할 때의 결과값이다. here,

is the first image of high quality, and P(

) is a high-quality sample extracted from the first image,

is a down-scaled sample of the extracted high-definition sample, and G(

) is a pseudo-high-definition sample in which the image quality of the down-scaled sample is improved, and D(G(

)) is a result value when the discriminator 220 determines the authenticity of the similar high-definition sample.

)는 생성자(210)에 의해 생성된 유사 고화질 샘플이 판별자(220)를 속이도록 학습하는데 사용되는 함수로서 기울기 값이 사라지는 문제를 최소화할 수 있다. Here, the first loss function (

) is a function used to learn that the similar high-definition sample generated by the generator 210 deceives the discriminator 220 , and can minimize the problem that the gradient value disappears.

[Equation 3]

)는 제 1 이미지로부터 추출된 고화질 샘플(P(

))과 생성자(210)에 의해 생성된 유사 고화질 샘플(G(

)) 간의 기학습된 네트워크의 19 계층함수인 f 함수로부터 추출된 특징 벡터 간의 거리가 최소화되도록 학습하는데 사용된다. Here, the second loss function (

) is a high-quality sample extracted from the first image (P(

)) and the pseudo-high-definition sample G(

))) is used to learn to minimize the distance between feature vectors extracted from the f function, which is a 19-layer function of the pre-trained network.

The discriminator 220 may learn to determine whether the high-quality similar sample generated by the generator 210 is an actual high-definition sample or a generated similar high-definition sample based on a preset discrimination loss function. The discriminator 220 may minimize a problem in which the gradient value disappears by using Least Square Generative Adversarial Networks (LSGAN). Here, the preset discriminant loss function may be expressed as [Equation 4].

[Equation 4]

Referring to FIG. 4 , the image quality improving unit 120 inputs the second image 401 to the learned adversarial generating neural network model to improve the image quality of the second image 401 to improve the image quality of the second image 403 . ) can be created.

Meanwhile, those skilled in the art will fully understand that the model generating unit 100 , the learning unit 110 , and the image quality improving unit 120 may be implemented separately, or one or more of them may be integrated.

An embodiment of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The description of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. .

10: Image quality improvement device
100: model generation unit
110: study department
120: image quality improvement unit

Claims (3)

In the image quality improvement apparatus,
a model generator for generating an adversarial generated neural network model including a generator for generating a similar high-definition sample from the first image and a discriminator for determining whether the similar high-definition sample is authentic;
the generator extracts samples of at least a portion of the first image, down-scales the samples, and improves the quality of the down-scaled samples to generate the pseudo-high-definition sample, and the discriminator determines the authenticity of the pseudo-high-definition sample. a learning unit that trains the adversarial generative neural network model to determine whether or not; and
An image quality improvement unit for improving the quality of the second image by inputting a second image to the learned adversarial generative neural network model
including,
The learning unit trains the generator to deceive the discriminator by generating a similar high-quality sample similar to the sample by the generator using a first loss function,
The first loss function is

which is an image quality improvement device.
The method of claim 1,
The learning unit trains the generator to minimize the distance between the feature vector of the sample and the feature vector of the similar high-definition sample using a second loss function,
The second loss function is

which is an image quality improvement device.
3. The method of claim 2,
In the first loss function and the second loss function,

denotes the first image, and P(

) represents the sample extracted from the first image,

denotes a down-scaled sample of the extracted sample, and G (

) represents a similar high-definition sample in which the picture quality of the down-scaled sample is improved, and D(G(

)) represents a result value when the discriminator determines whether the similar high-definition sample is authentic or not.

Images