KR102525181B1 - System for correcting image and image correcting method thereof - Google Patents

Abstract

An image correction method of the present invention includes the steps of performing a pre-processing process on an original image to generate a mask image including only erased areas in the original image; predicting an image to be synthesized into the erased region in the mask image using generative adversarial networks; and generating a new image by combining the predicted image with the original image in the original image.

Description

Image correction system and image correction method thereof {SYSTEM FOR CORRECTING IMAGE AND IMAGE CORRECTING METHOD THEREOF}

The present invention relates to techniques for modifying images, and more particularly to techniques for modifying facial images.

Recently, the number of people who share various information such as images through SNS is increasing, and according to this trend, interest in image editing programs or applications is increasing.

A conventional image editing program (or image editing tool) or an application related thereto mainly modifies an image by adjusting pixel values of the image. Realistically corrected images are determined by the user's proficiency in handling image editing programs.

Therefore, when a general user with little professional knowledge or experience in image editing programs modifies an image using an image editing program, the result is likely to be an unrealistic and awkward image.

Regarding image editing, there is also a method of editing a 3D model obtained by performing 3D modeling on an image using an engine (hereinafter referred to as a 3D model engine). However, this method requires a 3D model engine that implements a 3D model, and these 3D model engines are diverse, and the degree of completion of the result or the knowledge required for completion varies greatly depending on the 3D model engine used.

In addition, there is a limit that the user must be fully aware of how to use each 3D model engine. That is, in the existing methods, it is very difficult to realistically modify an image only with a simple image input.

An object of the present invention is to provide a facial image correction system and method capable of providing a realistic composite image more easily and quickly than conventional methods using a neural network trained in advance using facial images and user input.

That is, the present invention proposes a system and method that allow a user to easily and intuitively modify a face image and obtain a realistic result. It is an object of the present invention to enable anyone to easily modify a face image without requiring specialized knowledge or experience to obtain a realistic result.

The foregoing and other objects, advantages and characteristics of the present invention, and methods of achieving them will become clear with reference to embodiments described below in detail in conjunction with the accompanying drawings.

An image correction method according to an aspect of the present invention for achieving the above object includes performing a preprocessing process on an original image to generate a mask image including only erased areas in the original image; predicting an image to be synthesized into the erased region in the mask image using generative adversarial networks; and generating a new image by combining the predicted image with the original image in the original image.

An image retouching system according to another aspect of the present invention performs a pre-processing process on an original image to obtain a mask image including only a region erased by a user input in the original image, and a region erased by the user input. a pre-processing unit generating a sketch image including only the sketched shape and a color image including only the color applied to the erased area by the user input;

Using generative adversarial networks, an image to be synthesized in the erased area is predicted from the mask image, the sketch image, and the color image, and the predicted image is synthesized in the erased area to generate the original image. an image generating unit generating a new image from; and a display unit displaying the new image.

An image correction method according to another aspect of the present invention includes training a generative adversarial network composed of a generative neural network and a discriminant neural network in an adversarial relationship with each other; storing the generative neural network for which learning has been completed in a storage unit; From the original image through a pre-processing process, a mask image including only the area erased by the user input in the original image, a sketch image including only the shape sketched in the area erased by the user input, and the user input generating a color image containing only the color painted on the erased area; predicting an image to be synthesized in the erased area from the mask image, the sketch image, and the color image using a generative neural network stored in the storage unit; and generating a new image from the original image by combining the predicted image with the erased region.

According to the present invention, by modifying a face image using an adversarial neural network, a face image can be quickly and conveniently created in a manner desired by a user without specialized knowledge or experience of a separate image tool.

Existing image correction programs have separate tools for making the face thinner or the eyes bigger, and a lot of user experience is required to use them well.

However, the system provided by the present invention does not require a separate image tool, and can modify a face image in a desired direction by using a mask, sketch, or color as input information.

1 is a block diagram of a face image correction system according to an embodiment of the present invention;
2 is a diagram for explaining a plurality of input images input from the pre-processing unit shown in FIG. 1 to the image generating unit;
3 to 7 are diagrams illustrating a screen configuration of a user interface according to an embodiment of the present invention.
8 is a diagram showing the overall network structure of generative adversarial networks (GANs) applied to the present invention.
9 is a flowchart illustrating an image correction method according to an embodiment of the present invention.
10 to 12 are diagrams illustrating results of correcting an original image according to the image correction method of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are only illustrated for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention It can be embodied in many forms and is not limited to the embodiments described herein.

Embodiments according to the concept of the present invention can apply various changes and can have various forms, so the embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosures, and includes modifications, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named a second component, Similarly, the second component may also be referred to as the first component.

Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these examples. Like reference numerals in each figure indicate like elements.

1 is a block diagram of an image correction system according to an embodiment of the present invention.

Referring to FIG. 1 , the image correction system 100 according to an embodiment of the present invention can quickly and conveniently modify an image in a desired manner by a user without professional knowledge or sufficient experience in tools such as an image correction program.

The image correction system according to the present embodiment is limited to correcting a face image. Therefore, hereinafter, the image correction system is referred to as a 'face image correction system'. However, the present invention is not limited to face image correction, and can be applied to all types of image correction related to vehicle design, architectural design, home appliance design, and the like.

In order to design a system that allows a user to easily and quickly correct a face image, the face image correction system 100 includes an input unit 110, an anterior unit 120, an image generator 130, a display unit 140, and a storage unit. (150) and a learning unit (160).

input unit 110

The input unit 110 is a component that transmits user input to the layer creation unit 120, and hardware means such as a keyboard, mouse, touch pad, and touch panel and software means programmed to work with these hardware means (hereinafter referred to as 'user interface'). ').

The user can perform the following tasks through the input unit 110.

- Erase the area the user wants to correct in the original face image (10).

- An operation of sketching a shape (or shape) that the user wants to modify within an erased area of the original image 10. Here, the sketch means the level at which the shape can be recognized, and the expert level sketch is unnecessary.

- Coloring the sketched shape with the color the user wants to modify

According to the embodiment, the region to be corrected by the user includes an eye region, an ear region, a mouth region, a nose region, a hair region, and the like in the original face image.

According to the embodiment, the shape (or form) that the user wants to modify includes glasses worn in the eye area, earrings worn in the ear area, lip shape, hairstyle, and the like.

According to an embodiment, the color that the user wants to modify includes eye color, earring color, lip color, hair color, and the like.

The input unit 110 receives a first input value representing a region erased from the original image, a second input value representing a shape (or shape) sketched in the erased region, and a user input value corresponding to the above operations. A third input value representing a color to be corrected is generated and inputted to the preprocessing unit 120 .

Pre-processing unit 120

The pre-processing unit 120 pre-processes the original face image according to the first to third input values input from the input unit 110 and generates a plurality of pre-processed input images. The generated input images are used as information input to the image generating unit 130 .

2 is a diagram for explaining a plurality of input images input from a pre-processing unit to an image generating unit according to an embodiment of the present invention.

Referring to FIG. 2 , a plurality of input images include an original face image 11 including an area erased by a user according to the first input value, and a mask image including only the area erased according to the first input value. image) 13, a sketch image 15 representing only a shape sketched in the erased area by the user according to the second input value, and a color image including only colors appearing in the erased area according to the third input value ( 17) and a noise image 19 representing only Gaussian noise of the erased area.

image generator (130)

Referring back to FIG. 1 , the image generator 130 analyzes the plurality of input images input from the pre-processor 120 using a neural network trained in advance, and according to the analysis result, the original face image ( 10) predicts a realistic image that the user intends to combine with the erased area, and automatically synthesizes the predicted realistic image with the erased area of the original face image 10 .

According to an embodiment, the pretrained neural networks may be generative adversarial networks (GANs). Accordingly, the image generator 130 modifies the original face image using generative adversarial networks (GANs).

Generative adversarial networks (GANs) are deep neural network techniques introduced in a 2014 paper by researchers including Ian Goodfellow and Yoshua Bengio of the University of Montreal.

A detailed description of generative adversarial networks (GANs) is replaced by the above paper, and in this specification, only the brief operating principle of generative adversarial networks (GANs) and modified and changed parts to be applied to the present invention will be described.

Generative adversarial networks (GANs) are deep neural networks composed of a generative neural network (162), which can be called a generator, and a discriminative neural network (164, discriminative neural network), which can be called a discriminator. have a structure

The generative neural network 162 generates a new image to be transmitted to the discriminant neural network 164, and the discriminant neural network 164 determines whether the new image input from the generative neural network 162 is authentic.

The generative neural network 162 generates a new image, the discriminant neural network 164 determines the authenticity of the new image input from the generative neural network 162, and uses the discrimination result as an input of the generative neural network 162. give feedback

At this time, the generative neural network 162 learns the image generation process so that the discriminant neural network 164 determines the new image it has created as a real image, and conversely, the discriminant neural network 164 is a generative neural network. The image discrimination process is learned to discriminate the new image input from (162) as a fake image.

As such, the generative neural network 162 and the discriminant neural network 164 are trained through opposing objective functions or loss functions like a zero-sum game. That is, the generative neural network 162 and the discriminant neural network 164 learn and evolve to maintain an adversarial relationship with each other.

The image generator 130 uses only the generative neural network 162 for which learning has been completed among the generative neural network 162 and the discriminant neural network 164 to generate a realistic image synthesized in the deleted area of the original face image.

storage unit 150

The storage unit 150 stores the generative neural network 162 used by the image generator 130 . The generative neural network 162 may be replaced by the term generative algorithms. Similarly, discriminant neural networks 164 may be replaced by the term Discriminative Algorithms. The storage unit 150 may be implemented with volatile memory and non-volatile memory.

learning department (160)

The learning unit 160 trains generative adversarial networks (GANs), that is, the generative neural network 162 and the discriminant neural network 164, using a large amount of training data collected in advance.

Here, the large-capacity training data includes a large-capacity training face image, a large-capacity training face image configured to include erased areas at different locations, a large-capacity training mask image representing only the erased area, and various shapes in the erased area. It includes a large-capacity sketch image for training, a large-capacity color image for training in which different colors are filled in the erased area, and a large-capacity noise image.

Since the image generator uses only the generative neural network 162 among the learned generative neural network 162 and the discriminant neural network 164, the learning unit 160 uses the learned generative neural network 162 and the discriminant neural network ( By storing the generative neural network 162 of 164 in the storage unit 150, the image generator 130 can use the generative neural network 162 stored in the storage unit 150.

display unit 140

The display unit 140 is configured to display the image generated by the image generator 130, and a realistic image to be synthesized by the user is automatically synthesized in the erased area according to the user input within the original face image 10. Display composite image. The display unit 140 may be an LCD, LED, OLE, or the like.

Also, the display unit 140 displays the screen configuration of the user interface.

3 to 7 are diagrams illustrating a screen configuration of a user interface according to an embodiment of the present invention.

The user interface provides the user with an environment for generating input necessary for correcting the original face image, that is, input information input to the image generator.

To this end, the screen configuration of the user interface includes an area 31 displaying an original face image and a progress process for the original face image, and an area 32 displaying a modified result (composite image).

In addition, the screen configuration of the user interface includes a plurality of icon-shaped buttons 33, 34, 35, 36, and 37 related to generating input information.

First, referring to FIG. 3 , a button 33 provides a function of loading an original image. When the user touches or clicks the button 33, the original image is displayed in the area 31.

Referring to FIG. 4 , a button 34 provides a function for the user to erase a region to be modified in the original face image 10 . For example, when the user touches or clicks the button 34, an eraser-shaped icon tool is created, and a part to be corrected in the original image 10 is erased using the eraser-shaped icon tool. FIG. 4 shows a case where the user's original face image 10 is to be corrected in an eye area, a nose area, and a lip area.

Referring to FIG. 5 , a button 35 provides a function for the user to sketch a shape to be corrected in an area erased from the original face image 10 . For example, when the user clicks or touches the button 35, a pen-shaped icon tool is created, and a shape to be corrected is sketched in the area erased from the original face image 10 using the pen-shaped icon tool. . At this time, the sketch means the level at which the shape can be recognized, and the expert level sketch is unnecessary.

Referring to FIG. 6 , a button 36 provides a function of coloring a sketched shape with a color the user wants to modify. For example, when the user clicks or touches the button 36, an icon tool in the form of a brush is created, and the sketched shape or its surroundings is painted with a color determined by the user using the icon tool in the form of a brush. 6 illustrates a case in which the color of the pupil is painted in a color determined by the user.

Referring to FIG. 7 , button 37 provides a function of showing the modified result. When the user clicks or touches the button 37, a synthesized image in which a realistic image that the user wants to synthesize is automatically synthesized is displayed in the area 32 erased according to the user input within the original face image 10. do. FIG. 7 shows an example in which the nose shape image, the lip shape image, and the pupil color image corrected in the image generation process of the generative neural network 162 are combined with the original face image.

In FIG. 7 , an example in which the generative neural network 162 generates a synthesized image using all input information generated according to a mask operation (an operation of erasing an area that a user wants to correct in an original face image), a sketch operation, and a color operation , but not all input information is required to generate a synthesized image.

For example, the generative neural network 162 may generate a synthesized image using only one input information generated according to a mask operation, that is, a mask image. This is because the generative neural network 162 is learned using all large-capacity training data, that is, large-capacity mask images, large-capacity sketch images, and large-capacity color images.

This also means that the mask image indicating the location of the region to be corrected by the user in the original face image 10 must be used as an input to the generative neural network 162 . That is, the mask image can be interpreted as minimum information notifying the generative neural network 162 of the user's intention to modify.

Of course, the greater the amount of information input to the generative neural network 162, the higher the probability of generating a modified face image that most closely matches the user's intention to modify.

8 is a diagram showing the overall network structure of generative adversarial networks (GANs) applied to the present invention.

As described above, in order to train the generative neural network 162, the entire network (GANs) composed of the generative long neural network 162 and the discriminant neural network 164 must be trained together.

When learning is completed, only the generative neural network 162 is used to modify the face image, so in order to improve the speed of the generative neural network 162 and increase the realism of the modified face image, generative adversarial networks (GANs) are It can be configured with a structure as shown in 8.

As shown in FIG. 8, the generative neural network 162 is composed of a U-net structure and includes gated convolution layers, dilated gated convolution layers, and deconvolution. It includes deconvolution layers. Each convolution layer is shown as a hexahedron shape with different sizes according to the number of channels.

The generative neural network 162 is characterized by using a gated convolution layer instead of a general convolution layer. A general convolution layer outputs a different feature value for a feature value input from a previous convolution layer. In contrast, the gate convolution layer outputs other feature values for features input from the previous gate convolution layer and feature values for the mask image. That is, there is a difference in that a general convolution layer outputs one data and a gate convolution layer outputs two data. One of the two pieces of data output from the gate convolution layer is a feature value of a mask image, and the feature value of the mask image is input to another non-adjacent gate convolution layer. In FIG. 8 , a dotted line arrow indicates an input to another gate convolution layer in which the feature values of the mask image output by the current gate convolution layer are not adjacent.

The discriminant neural network 164 has a patchGAN form consisting of a gated convolution layer to which spectral normalization is applied. The process of training the discriminant neural network 164 is the same as the method of training general generative adversarial networks (GANs) for a dataset.

However, a loss variable (Loss) used for learning of the discriminant neural network 164 is different from a general loss (Loss). Here, loss means a difference between an original face image and a new face image generated by the generative neural network 162 .

The parameters (L _G ) used to train the generative neural network 162 according to an embodiment of the present invention are as shown in Equation 1 below, and the parameters (L _D ) used to learn the discriminant neural network 164 are as follows: It is the same as Equation 2 of

The parameter L _G is a loss for learning a layer of the generative neural network 162 . Here, as shown in FIG. 8, the layers of the generative neural network 162 include a plurality of gated convolution layers, a plurality of dilated gated convolution layers, and a plurality of gated convolution layers. It includes a deconvolution layer of

The parameter L _D is a loss for learning a layer of the discriminant neural network 164 . Here, a layer of the discriminant neural network 164 includes a plurality of spectral normalization (SN) convolutional layers.

In Equation 1, L _{per -pixel} is equal to Equation 3 below.

는 이미지 크기에 따른 총 픽셀 수로서, 예컨대, 786432(= 512 x 512 x 3)일 수 있다.Here, M is a pixel value of one channel representing an erased area included in the mask image, and may be, for example, '1'. At this time, each pixel value of the area other than the erased area in the mask image is '0'. I _gen is a new face image generated by the generative neural network 162 by receiving an original face image having an erased region, a mask image having only the erased region, a sketch image, a color image, and a noise image, that is, a new face image erased from the original face image. An image in which an area is filled with other images, and I _gen may be a 3D vector value expressing a new face image generated by the generative neural network 162 in a 3D vector space. I _gt means an original face image without an erased region, and I _gt may be a 3D vector value expressing the original face image in a 3D vector space. L _{per -pixel} is a parameter that calculates the distance L1 between the new face image created by the generative neural network and the original face image. The distance L is the distance between a pixel of the new face image and a pixel of the original face image corresponding to the pixel. α is a weight for enhancing (reducing) the loss of the erased area in the original face image. ⊙ is a symbol representing element-wise multiplication,

is the total number of pixels according to the image size, and may be, for example, 786432 (= 512 x 512 x 3).

In Equation 1, L _percept is equal to Equation 4 below.

는 L1 거리에 대한 손실(loss)로서, 모든 픽셀 차이의 절대값의 합이다. I_comp는 지워진 영역을 갖는 원본 얼굴 이미지와 생성적 신경망(162)에 의해 생성된 새로운 얼굴 이미지(I_gen)에서 상기 지워진 영역에 대응하는 부분만을 추출한 이미지를 합성한 합성 이미지이다. 즉, I_gen는 생성적 신경망(162)의 예측 및/또는 추론 과정을 통해 생성된 이미지를 의미하는 것인 반면, I_comp는 예측 및/또는 추론 과정이 아니라 지워진 영역을 갖는 원본 이미지와 생성적 신경망(162)에 의해 생성된 이미지(I_gen)로부터 추출된 이미지(상기 지워진 영역에 대응하는 이미지)를 합성하는 합성 과정을 통해 생성된 이미지를 의미한다. I_comp는 3차원 벡터 공간에서 표현되는 3차원 벡터 값일 수 있다.L _percept is one of the losses used as style-loss. θ _q denotes a feature of the q-th layer of a generative neural network learned for image classification on a large training data set. That is, L _percept is a loss for a feature calculated in units of pixels using an existing learned network. In N_θ _q (I_gt), N is the total number of pixels, and θ _q is the q-th feature map of an artificial neural network that has learned a large amount of images for image classification, which is different from generative neural networks. That is, N_θ _q (I_gt) is the total number of pixels of the qth feature map obtained by passing the original face image through the image classification artificial neural network.

is the loss for the L1 distance and is the sum of the absolute values of all pixel differences. I _comp is a composite image obtained by synthesizing an original face image having an erased region and an image obtained by extracting only a part corresponding to the erased region from the new face image I _gen generated by the generative neural network 162 . That is, I _gen denotes an image generated through the prediction and/or inference process of the generative neural network 162, whereas I _comp is not a prediction and/or inference process but an original image having an erased area and a generative It refers to an image generated through a synthesis process of synthesizing an image (an image corresponding to the erased area) extracted from an image (I _gen ) generated by the neural network 162 . I _comp may be a 3D vector value expressed in a 3D vector space.

In Equation 1, Lstyle (I _gen ) is equal to Equation 5 below.

L _style is the loss of the feature of the qth layer of the network learned for image classification on a large face data set. Gram matrix can be utilized to calculate this L _style . C _q is the number of channels in the q-th layer, N _q is the total number of pixels in the q-th layer, and G _q represents the Gram matrix value of the q-th layer.

Additionally, variance loss is used to train the generative neural network 162. The variance loss represents the difference between the original face image and the new face image obtained by the discriminant neural network 164 forcibly moving the mode pixel in the new face image input from the generative neural network 162 by 1 pixel. It is a loss. This variance loss is shown in Equation 6 below.

The variance loss includes L _{tv -col and} L _{tv -row} , and in Equation 6 above, the subscript comp means an image in which a region erased from an original face image is filled with an image according to a user input. N _comp is the total number of pixels of the new face image generated by the generative neural network 162 .

The generative neural network 162 is trained to be robust against blurring using the above variance loss.

Finally, a gradient penalty term (L _GP ) is added as follows to prevent the discriminant neural network 164 from converging on the training data. Here, the meaning of preventing the discriminant neural network 164 from converging on training data means that the discriminant neural network 164 should not be trained to a higher level than the generative neural network 162 . That is, it means that the generative neural network 162 and the discriminant neural network 164 must be trained at the same level to maintain an adversarial relationship with each other.

As described above, when the process of learning the generative neural network 162 and the discriminant neural network 164 in the direction of decreasing the parameter L _G and the parameter L _D is completed, thereafter, the image generator 130 generates a A new face image is generated using only the neural network 162 .

Since the generative neural network 162 is light in size, it takes less than 2 seconds to generate a new face image based on a general CPU.

9 is a flowchart illustrating an image correction method according to an embodiment of the present invention.

Referring to FIG. 9 , in S910, a learning process of training the generative adversarial neural network 160 is performed. This learning process is a process of learning the generative neural network and the discriminant neural network to maintain an adversarial relationship with each other. As will be described below, when the learning of the generative neural network 162 and the discriminant neural network 164 is completed, an image to be synthesized with the original image is predicted using only the generative neural network, and the predicted image and the original image are synthesized. To train the generative neural network 162, a large amount of previously collected training data is used. A large amount of training data includes original training images, training mask images including only the erased area, training sketch images in which various shapes are sketched in the erased area, and training color images filled with various colors in the erased area. include them Additionally, a noise image representing noise of the erased area may be further included. In order to train the generative neural network 162 to maintain an adversarial relationship with the discriminant neural network 164, a loss (L _G ) output from the discriminant neural network 164 may be used. The process of learning the generative neural network 162 is a process of learning in a direction of reducing the loss (L _G ). In this case, the loss (L _G ) is a pixel difference value (I _gen -I _gt ). A description thereof is replaced with the description of Equations 1 and 3 above. In addition, the process of learning the generative neural network 162 is a pixel difference value (I _gen ) between the pixel value (M) of the erased area in the training mask image included in the large amount of training data and the original image and the new image This may be a process of learning a gated convolution layer included in the generative neural network using -I _gt .

Subsequently, in S920, when the learning process for the generative adversarial neural network 160 is completed, that is, when the learning of the generative neural network 162 and the discriminant neural network 164 is completed, the learning-completed generative neural network 162 A process of storing in the storage unit 150 is performed. By doing this, the image generator 130 can access the generative neural network 162 .

Subsequently, in S930, a pre-processing process is performed on the original image 10, and through this pre-processing process, information input from the original image 10 to the generative neural network 162 on which the learning is completed is generated. Information input to the generative neural network 162 includes a mask image including only a region erased by a user input in the original image, a sketch image including only a shape sketched in the region erased by the user input, and the and a color image including only the color painted on the area erased by a user input. In this case, the information input to the generative neural network 162 may consist of only a mask image including only a region erased by a user input in the original image. As described above, according to the present invention, a part of an original image can be modified according to the user's intention by simply inputting only the mask image containing only the location information of the region erased by the user to the generative neural network 162 .

Next, in S940, the image generator 130 analyzes the mask image, the sketch image, and the color image using the generative adversarial network that has been learned, that is, the generative neural network 162 that has been learned. , a process of predicting an image to be synthesized in the erased area in the original image according to the analysis result is performed.

Subsequently, in S950, the image generator 130 synthesizes the predicted image with the erased area in the original image to generate a new image, and the display unit 140 displays the generated new image to the user. Provided.

10 to 12 are diagrams illustrating composite images generated according to the image correction method of the present invention.

As shown in (A) and (B) of FIG. 10 , according to the present invention, a new image can be automatically generated by correcting the shape of the mouth and eyes in the original face image. Also, as shown in (C) of FIG. 10 , a new image may be automatically generated by deleting the glasses from the original face image. The important point here is that the shape sketched in the erased area to create the free-form input of the generative neural network does not require expert-level sketching. That is, since the present invention uses a generative neural network trained in advance using a large amount of training data, it is possible to infer the image to be synthesized with the original face image only with the location information of the area erased from the mask image and the shape sketched at the level of the general public. Predictable. This means that original face images can be realistically modified using only simple image inputs such as mask images, sketch images or color images.

Further, according to the present invention, as shown in (A) of FIG. 11, part of the hair style may be corrected, or as shown in (B), all of the hairstyle may be corrected.

In addition, according to the present invention, as shown in (A) of FIG. 12, a partial region of the ear region is erased from the original face image, and an earring shape simply sketched by the user is input to the erased region as input information of the generative neural network. When configured, it is possible to automatically generate an image in which the original face image and the earring image are realistically synthesized.

In addition, as shown in (B) of FIG. 12 , it will be possible to modify the earrings included in the original face image to earrings of a different shape.

The image correction system described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components.

For example, components such as the pre-processing unit 120, the image generating unit 130, and the learning unit 160 described in the embodiments are, for example, a processor, a controller, an arithmetic logic unit (ALU), a graphic processor (GPU), digital signal processor (digital signal processor), microcomputer, FPGA (field programmable gate array), PLU (programmable logic unit), may be implemented as a microprocessor.

Also, the image modification system may execute an operating system (OS) and one or more software applications running on the operating system. The image modification system may also access, store, manipulate, process and create data in response to execution of the software.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (16)

generating a mask image including only erased areas in the original image by performing a preprocessing process on the original image;
predicting an image to be synthesized in the erased area in the mask image using a generative adversarial network including a generative neural network and a discriminant neural network; and
generating a new image by combining the predicted image with the erased region in the original image;
As a loss function for training the generative neural network,
The loss function is
The first feature map obtained from the image generated by the generative neural network based on an artificial neural network different from the generative neural network, the original image having the erased area, and the erased image extracted from the new image generated by the generative neural network. When defining a second feature map obtained by synthesizing an image corresponding to a region based on the artificial neural network and a third feature map obtained by using the original image based on the artificial neural network, the first feature map and the first feature map are defined. 3 An image correction method calculated based on a loss for a distance between feature maps and a loss for a distance between the second feature map and the third feature map. In paragraph 1,
The predicting step is
and predicting the image to be synthesized using the generative neural network. In paragraph 1,
Prior to generating the mask image, the step of learning the generative adversarial neural network using a large amount of training data is further included,
The large amount of training data,
Original images for training, training mask images including only the erased area, training sketch images in which various shapes are sketched in the erased area, and training color images filled with various colors in the erased area How to edit images. In paragraph 1,
Prior to generating the mask image, the step of learning the generative adversarial neural network using a large amount of training data is further included,
The step of learning the generative adversarial neural network,
learning a generative neural network using the large amount of training data; and
Training a discriminative neural network to maintain an adversarial relationship with the generative neural network;
The predicting step is
and predicting the image to be synthesized using the generative neural network for which learning has been completed. In paragraph 4,
Training the generative neural network,
Learning the generative neural network using a loss (L _G ) output from the discriminant neural network;
The loss (L _G ) is,
Image correction method comprising a pixel value (M) of an erased area in the training mask image included in the large amount of training data and a pixel difference value (I _gen -I _gt ) between the original image and the new image. . In paragraph 4,
Training the generative neural network,
The generative neural network using a pixel value (M) of an erased area in the training mask image included in the large amount of training data and a pixel difference value (I _gen -I _gt ) between the original image and the new image An image correction method that is a step of learning a gated convolution layer included in. By performing a pre-processing process on the original image, a mask image including only the area erased by the user input in the original image, a sketch image including only the shape sketched in the area erased by the user input, and the user input a pre-processing unit generating a color image including only the color painted on the erased area by;
Using a generative adversarial network including a generative neural network and a discriminant neural network, an image to be synthesized in the erased area is predicted from the mask image, the sketch image, and the color image, and the predicted image an image generating unit generating a new image from the original image by synthesizing the erased area with the erased region; and
A display unit for displaying the new image;
As a loss function for training the generative neural network,
The loss function is
The first feature map obtained from the image generated by the generative neural network based on an artificial neural network different from the generative neural network, the original image having the erased area, and the erased image extracted from the new image generated by the generative neural network. When defining a second feature map obtained by synthesizing an image corresponding to a region based on the artificial neural network and a third feature map obtained by using the original image based on the artificial neural network, the first feature map and the first feature map are defined. 3 An image correction system calculated based on a loss for a distance between feature maps and a loss for a distance between the second feature map and the third feature map. In paragraph 7,
The image generator,
and predicting the image to be synthesized using the generative neural network. In paragraph 7,
a learning unit for learning the generative adversarial network using a large amount of training data; and
Further comprising a storage unit for storing the generative adversarial network for which learning has been completed by the learning unit,
The large amount of training data,
Original images for training, training mask images including only the erased area, training sketch images in which various shapes are sketched in the erased area, and training color images filled with various colors in the erased area Image correction system. In paragraph 9,
The learning unit,
Training the generative adversarial network including a generative neural network and a discriminative neural network to maintain an adversarial relationship with each other;
the storage unit,
An image correction system wherein only the generative neural network for which learning has been completed is stored. In paragraph 10,
The learning unit,
Learning the generative neural network in the direction of reducing the loss (L _G ) output from the discriminant neural network;
The loss (L _G ) is,
An image correction system comprising a pixel value (M) of an erased area in the training mask image included in the large amount of training data and a pixel difference value (I _gen -I _gt ) between the original image and the new image. . In paragraph 10,
The learning unit,
The generative neural network using a pixel value (M) of an erased area in the training mask image included in the large amount of training data and a pixel difference value (I _gen -I _gt ) between the original image and the new image An image correction system that trains a gated convolution layer included in. Learning a generative adversarial network composed of a generative neural network and a discriminant neural network in an antagonistic relationship with each other;
storing the generative neural network for which learning has been completed in a storage unit;
From the original image through a pre-processing process, a mask image including only the area erased by the user input in the original image, a sketch image including only the shape sketched in the area erased by the user input, and the user input generating a color image containing only the color painted on the erased area;
predicting an image to be synthesized in the erased area from the mask image, the sketch image, and the color image using a generative neural network stored in the storage unit; and
generating a new image from the original image by combining the predicted image with the erased region;
As a loss function for training the generative neural network,
The loss function is
The first feature map obtained from the image generated by the generative neural network based on an artificial neural network different from the generative neural network, the original image having the erased area, and the erased image extracted from the new image generated by the generative neural network. When defining a second feature map obtained by synthesizing an image corresponding to a region based on the artificial neural network and a third feature map obtained by using the original image based on the artificial neural network, the first feature map and the first feature map are defined. 3 An image correction method calculated based on a loss for a distance between feature maps and a loss for a distance between the second feature map and the third feature map. In paragraph 13,
The learning step is
A step of learning a generative neural network and a discriminative neural network to maintain an adversarial relationship with each other,
Training the generative neural network,
In the generative neural network, using the pixel value (M) of the erased area in the training mask image included in the large amount of training data and the pixel difference value (I _gen -I _gt ) between the original image and the new image An image correction method that is a step of learning a gated convolution layer included. In paragraph 13,
In the step of predicting an image to be synthesized in the erased area,
The image to be synthesized is
The image correction method is a realistic image predicted according to a shape sketched in the erased area. In paragraph 13,
Wherein the original image is a face image.

Images