WO2022260392A1

WO2022260392A1 - Method and system for generating image processing artificial neural network model operating in terminal

Info

Publication number: WO2022260392A1
Application number: PCT/KR2022/007999
Authority: WO
Inventors: 장한힘; 권민수; 이명제
Original assignee: 주식회사 에너자이
Priority date: 2021-06-10
Filing date: 2022-06-07
Publication date: 2022-12-15
Also published as: KR20220166712A

Abstract

The present invention relates to a method and system for generating an image processing artificial neural network model operating in a terminal, and the objective of the present invention is to provide a method for training a small artificial neural network model having an excellent performance operating in a smart terminal. The present invention for achieving such an objective is characterized by a method by which an electronic device generates a second artificial neural network model that removes noise, the second artificial neural network model being applicable to a smart terminal, the method comprising: an a step for generating a first query image and a second query image; a b step for training a first artificial neural network model on the basis of the first query image and the second query image; and a c step for extracting the second artificial neural network model from the first artificial neural network model and transmitting same to the smart terminal.

Description

Method and system for generating an image processing artificial neural network model operating in a terminal

The present invention relates to a method and system for generating an artificial neural network model operating in a terminal, and more particularly, to a method and system for generating a small artificial neural network model that removes noise from an image in a terminal without passing through a server.

As artificial intelligence technology develops, a method of processing image noise removal with an artificial neural network is commercialized.

However, an artificial neural network model performing a noise removal function generally requires high-specification computing resources because performance is excellent as the size increases. Therefore, artificial neural network models with excellent performance have a disadvantage that cannot be executed in widely used smart terminals.

The present invention is to solve the above problems, and an object of the present invention is to provide a method for generating a small artificial neural network model with excellent performance that operates in a smart terminal.

Another object of the present invention is to provide a process of learning a large artificial neural network model including a small artificial neural network model in order to ensure noise removal performance of the small artificial neural network model.

In order to achieve this object, the present invention provides a method for generating a second artificial neural network model in which an electronic device removes noise applicable to a smart terminal, including steps a of generating a first query image and a second query image; Step b of learning a first artificial neural network model based on the query image and the second query image, and step c of extracting a second artificial neural network model from the first artificial neural network model and transmitting the same to a smart terminal. do.

In addition, the present invention generates a first query image and a second query image, learns a first artificial neural network model based on the first query image and the second query image, and converts a second artificial neural network model from the first artificial neural network model. It is characterized by having an electronic device including a control module for extracting, a communication module for transmitting the second artificial neural network model to a smart terminal, and a storage module for storing the second artificial neural network model.

According to an embodiment of the present invention, there is an effect of generating a small artificial neural network model that operates in a smart terminal having lower computing specifications than a server.

In addition, according to an embodiment of the present invention, through the process of learning a large artificial neural network model including a small artificial neural network model, the performance of the small artificial neural network model can be guaranteed to the level of a large artificial neural network model operating in a server. there is

1 is a diagram showing the configuration of a system for learning an artificial neural network model for noise removal in a smart terminal according to an embodiment of the present invention.

2 is a diagram for explaining a process of learning an artificial neural network model for noise removal in a smart terminal according to an embodiment of the present invention.

3 is a flowchart illustrating a method of learning an artificial neural network model for noise removal in a smart terminal according to an embodiment of the present invention.

4 is a diagram for explaining in detail a learning method of a first artificial neural network model according to an embodiment of the present invention.

The above objects, features and advantages will be described later in detail with reference to the accompanying drawings, and accordingly, those skilled in the art to which the present invention belongs will be able to easily implement the technical spirit of the present invention. In describing the present invention, if it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

In the drawings, the same reference numerals are used to indicate the same or similar elements, and all combinations described in the specification and claims may be combined in any manner. And unless otherwise specified, it should be understood that references to the singular may include one or more, and references to the singular may also include plural.

Terms used herein are only for the purpose of describing specific exemplary embodiments and are not intended to be limiting. Singular expressions as used herein may also be intended to include plural meanings unless the context clearly dictates otherwise. The term “and/or,” “and/or” includes all combinations and any one of the associated listed items. The terms "comprises", "comprising", "including", "including", "having", "having" and the like are meant to be inclusive, and thus such terms shall be construed as having a recited feature, integer, Specifies steps, operations, elements, and/or components, and does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. The method steps, processes, and actions described herein should not be construed as requiring their performance in the specific order discussed or illustrated, unless such order of performance is specifically established. . It should also be understood that additional or alternative steps may be used.

In addition, each component may be implemented as a hardware processor, and the above components may be integrated and implemented as one hardware processor, or the above components may be combined with each other and implemented as a plurality of hardware processors.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a system for learning an artificial neural network model that operates in a smart terminal according to an embodiment of the present invention and performs a function of removing noise from an image.

Referring to FIG. 1, an electronic device 10 of the present invention is a device for learning an artificial neural network model and transmitting it to a smart terminal so that the smart terminal 20 can remove noise, and includes a control module 11 and a communication module. (13), a storage module 15 may be included.

The control module 11 may generate and learn a first artificial neural network model so that the first artificial neural network model can identify noise in the query image and remove it.

The control module 11 may utilize the training data set to learn the first artificial neural network model. The training data set according to an embodiment of the present invention includes a first query image with noise and a second query image that is the same as the first query image but without noise, and the control module 11 controls the first artificial query image. A first artificial neural network model may be created and trained through the first query image and the second query image so that the neural network model can accurately identify noise.

The control module 110 may extract a second artificial neural network model from the first artificial neural network model when it is determined that the first artificial neural network model has a certain level of accuracy or higher. The second artificial neural network model is a part of the first artificial neural network model, and is sufficiently operable even in a smart terminal 20 having low hardware performance due to less required computing resources compared to the first artificial neural network model.

The communication module 13 may transmit the second artificial neural network model to the smart terminal 20 .

The storage module 15 may include an image database storing a plurality of images for learning the first artificial neural network model and a model database storing the first artificial neural network model.

The smart terminal 20 may remove noise from the query image received from the user using the second artificial neural network model received from the electronic device 10, and specifically, the control module 21, the communication module 23, A storage module 25 may be included.

The control module 21 may remove noise from the image received from the user by using the second artificial neural network model received from the electronic device 10 .

The communication module 23 may receive an image from a user and receive a second artificial neural network model from the electronic device 10 .

The storage module 25 may store the second artificial neural network model.

2 is a diagram for explaining a process of learning an artificial neural network model for noise removal in a smart terminal according to an embodiment of the present invention. The first artificial neural network model 30 shown in FIG. 2 may be learned based on a training data set. The first artificial neural network model 30 shown in FIG. 2 may be stored in the storage module 15 shown in FIG. 1 and executed by the control module 11 .

First, the control module 11 may generate a training data set to learn the first artificial neural network model 30 . The control module 11 may generate a training data set using images stored in the image database of the storage module 15 .

Specifically, the control module 11 may generate a first query image and a second query image having a predetermined size using images stored in an image database.

The control module 11 may generate a first query image of a patch unit having a predetermined size from an image stored in an image database, and may generate a second query image by randomly generating noise in the first query image.

The control module 11 may generate various noises for the first query image by using a noise generation algorithm such as a Gaussian noise generation algorithm, and may generate a second query image in addition to the first query image.

The control module 11 may learn the first artificial neural network model 30 based on the first query image and the second query image. The first artificial neural network model 30 is a super network and may be composed of a target network 40 and a main network 50 having a coupling relationship with each other.

Since the target network according to an embodiment of the present invention is formed in a u-net structure, it may be composed of at least one pooling layer and an unpooling layer. The first artificial neural network model 30 generated by the noise removal system of the present invention is characterized in that it can use both microscopic and macroscopic features of the noise extraction target image as it includes a target network having a unitet structure as part of it.

In the first artificial neural network model 30, which is a super network in which the target network 40 and the main network 50 have a coupling relationship, the control module 11 uses the second query image as input data to generate noise of the second query image. can be extracted.

Specifically, the control module 11 may use the second query image as input data of the target network 40 and extract first noise for the second query image from the target network 40 .

The control module 11 may generate a third query image from which the first noise extracted from the second query image is removed through the target network 30 .

The control module 11 may use the second query image and the third query image as input data of the main network 50 and extract second noise for the second query image from the main network 50 .

The control module 11 may generate a fourth query image obtained by removing the second noise extracted from the second query image through the main network 50 .

The control module 11 may generate a third query image and a fourth query image by offsetting the first noise and the second noise in each of the second query images.

The control module 11 may calculate a first loss value by comparing the third query image extracted through the target network 40 with the first query image. The control module 11 may learn the target network 40 using the first loss value.

The control module 11 may calculate a second loss value by comparing the fourth query image extracted through the main network 50 with the first query image.

The control module 11 may learn the super network 30 using the first loss value and the second loss value. That is, the control module 11 may use both the first loss value and the second loss value to generate a loss value more effective for learning of the target network 40 included in the super network 30 .

In particular, the control module 11 may use both the first loss value and the second loss value in learning the super network 50, and may specifically assign a weight to each of the first loss value and the second loss value. have. As the noise removal system according to an embodiment of the present invention is applied through the target network 40 (the second artificial neural network model), a higher weight may be assigned to the first loss value for the target network 40 .

The control module 11 may use feature values between query images to calculate the first loss value and the second loss value.

Specifically, the control module 11 may use feature values of the first query image and the third query image to calculate the first loss value. The control module 11 may calculate a first loss value by subtracting feature values of the first query image and the third query image.

The control module 11 may use feature values of the first query image and the fourth query image to calculate the second loss value. The control module 11 may calculate a second loss value by subtracting feature values of the first query image and the fourth query image.

The control module 11 may set the difference between two query images to be compared as a loss value by subtracting the feature value of each query image, so that the difference between the two query images is minimized, that is, the loss value is the minimum value. The target network 40 and the super network 50 may be learned.

If the first loss value and the second loss value are greater than or equal to the preset first loss threshold, the control module 11 sets a weight for the second loss value to 0 and a weight for the first loss value to 1 so that the target Only network 40 can be learned. Accordingly, the performance of the target network 40 itself can be further improved.

The control module 11 may extract the target network 40 and set it as a second artificial neural network model when the first loss value and the second loss value are greater than or equal to a preset second loss threshold.

3 is a flowchart illustrating a method of learning an artificial neural network model for noise removal in a smart terminal according to an embodiment of the present invention. Hereinafter, a method of learning an artificial neural network model for noise removal will be described with reference to FIG. 3 . In the description of the method for learning the artificial neural network model for noise elimination, detailed embodiments overlapping with the system for learning the artificial neural network model for noise elimination described above may be omitted.

In step 100, the electronic device may generate a training data set. The electronic device may generate a training data set by using an image stored in an image database in order to generate and train the first artificial neural network model.

The electronic device may generate a first query image and a second query image having a preset size using images stored in an image database.

The electronic device may generate a first query image in a patch unit having a preset size from an image stored in an image database, and may randomly add noise to the first query image to generate a second query image.

The electronic device may generate various noises for the first query image by using a noise generation algorithm such as a Gaussian noise generation algorithm and generate a second query image in addition to the first query image.

In step 200, the electronic device may learn a first artificial neural network model based on a training data set including a first query image and a second query image. The electronic device may use a super network as a first neural network model, and the super network may include a target network and a main network having a coupling relationship with each other.

Referring to FIG. 4 , in step 210, the electronic device may extract the first noise for the second query image from the target network by using the second query image as input data of the target network.

In step 220, the electronic device may generate a third query image from which the first noise extracted from the second query image is removed through the target network.

In operation 230, the electronic device may use the second query image and the third query image as input data of the main network and extract second noise for the second query image from the main network.

In step 240, the electronic device may generate a fourth query image obtained by removing the second noise extracted from the second query image through the main network.

The electronic device may create a third query image and a fourth query image by offsetting the first noise and the second noise in each of the second query images.

In step 250, the electronic device may calculate a first loss value by comparing the third query image extracted through the target network with the first query image.

In step 260, the electronic device may calculate a second loss value by comparing the fourth query image extracted through the main network with the first query image.

In step 270, the electronic device may learn the target network and the super network using the first loss value and/or the second loss value. The electronic device may learn the target network using the first loss value and learn the super network using the first loss value and the second loss value.

In detail, the electronic device may use both the first loss value and the second loss value in order to generate a loss value more effective for learning the target network included in the super network.

In particular, the electronic device may learn the super network by assigning weights to each of the first loss value and the second loss value. As the noise cancellation system according to an embodiment of the present invention is applied through the target network, a higher weight may be assigned to the first loss value for the target network.

The electronic device may use feature values between query images to calculate the first loss value and the second loss value.

Specifically, the electronic device may use feature values of the first query image and the third query image to calculate the first loss value. The electronic device may calculate a first loss value by subtracting feature values of the first query image and the third query image.

The electronic device may use feature values of the first query image and the fourth query image to calculate the second loss value. The electronic device may calculate a second loss value by subtracting feature values of the first query image and the fourth query image.

The electronic device may set the difference between two query images to be compared as a loss value by subtracting the feature value of each query image, and the target network such that the difference between the two query images is minimized, that is, the loss value becomes the minimum value. and super networks.

When the first loss value and the second loss value are equal to or less than the preset first loss threshold, the electronic device learns only the target network by setting the weight for the second loss value to 0 and the weight for the first loss value to 1. can do. Accordingly, the performance of the target network itself can be further improved.

Returning to the description of FIG. 2 again, in step 300, the electronic device may extract a target network and set it as a second artificial neural network model. Specifically, when the first loss value and the second loss value are equal to or greater than a preset second loss threshold, the electronic device may extract the target network and set it as the second artificial neural network model.

In step 400, the electronic device may transmit the second artificial neural network model to the smart terminal.

When receiving the second artificial neural network model from the electronic device, the smart terminal may provide a noise removal service to the user through the small-sized second artificial neural network model. Specifically, when a query image is received from a user, the control module 21 of the smart terminal controls the operation of the second artificial neural network model based on the CPU, NPU or GPU.

The embodiments of the present invention disclosed in the present specification and drawings are only presented as specific examples to easily explain the technical content of the present invention and help understanding of the present invention, and are not intended to limit the scope of the present invention. It is obvious to those skilled in the art that other modified examples based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein.

Claims

A method for an electronic device to learn an artificial neural network model for noise removal in a smart terminal,

Step a of generating a first query image and a second query image;

b) learning a first artificial neural network model based on the first query image and the second query image; and

An artificial neural network learning method for noise removal comprising step c of extracting a second artificial neural network model from the first artificial neural network model and transmitting the second artificial neural network model to a smart terminal.
The method of claim 1, wherein step a,

generating a first query image having a predetermined size using an image stored in an image database;

An artificial neural network learning method for removing noise, comprising generating a second query image by adding random noise to a first query image.
The method of claim 2, wherein step b,

The first artificial neural network model is composed of a super network including a target network and a main network,

An artificial neural network learning method for removing noise, respectively, extracting first noise and second noise of a second query image through the target network and the main network.
According to claim 3,

extracting a first noise by inputting a second query image to the target network;

generating a third query image by removing the first noise from the second query image;

extracting a second noise by inputting a second query image and a third query image to the main network;

An artificial neural network learning method for removing noise, comprising generating a fourth query image by removing second noise from a second query image.
According to claim 4,

calculating a first loss value using the first query image and the third query image;

calculating a second loss value using the first query image and the fourth query image;

An artificial neural network learning method for noise removal comprising the step of learning the target network and the super network using a first loss value and a second loss value.
According to claim 5,

An artificial neural network learning method for denoising the target network and the super network by assigning weights to each of the first loss value and the second loss value.
The method of claim 6, wherein step c,

An artificial neural network learning method for denoising, in which a weight for a second loss value is set to 0 and only the target network is learned when the first loss value and the second loss value are equal to or less than a preset first loss threshold value.
The method of claim 6, wherein step c,

If the first loss value and the second loss value are less than or equal to a preset second loss threshold, setting the target network as a second artificial neural network model.
A control module generating a first query image and a second query image, learning a first artificial neural network model based on the first query image and the second query image, and extracting a second artificial neural network model from the first artificial neural network model. ; A communication module for transmitting the second artificial neural network model to the smart terminal; and a storage module for storing the second artificial neural network model; and

An artificial neural network learning system for noise removal comprising a smart terminal configured to receive a second artificial neural network model from the electronic device and remove noise from an image received from a user.