KR20220050504A - Appratus and method for converting a facial images - Google Patents

Abstract

The present invention relates to a GAN-based facial image conversion method. The method: generates a plurality of facial image learning data having various angles and distances by executing warping a source image of a first stage of converting scale; generates a facial image conversion module by deep learning the warped or scaled facial image learning data based on a generative adversarial neural network (GAN); and generates an input image similar to the trained image by inputting an input image to be converted into the facial image conversion module.

Description

FACIAL IMAGE CONVERSION DEVICE AND FACIAL IMAGE CONVERSION METHOD

The present invention relates to digital image processing technology, and more particularly, to an apparatus and method for converting a facial image to be similar to a desired image using a generative adversarial network (GAN).

Due to the development of mobile-related technologies, mobile devices in which various functions such as a camera, the Internet, and a display are implemented have been developed and used as necessities of people. These mobile devices are used in various ways to enjoy daily life.

Most representatively, a mobile device is utilized to perform a photo-related function. For example, the mobile device may be used to simply take a photo and upload the photo to one's own social network system (SNS) account. Recently, as social media has become deeply embedded in people's daily lives, the amount of time spent using photo-related functions on mobile devices is increasing.

Among the photos uploaded to social media, portrait images occupy a significant proportion. However, the conventional person image editing technology on a mobile device is at the level of filtering the entire person image. Some portrait image editing technologies allow the transformation of faces included in portrait images, but there are limitations in terms of user convenience due to difficulties in operation.

For example, Patent Document 1 (Korean Patent Application Laid-Open No. 2020-0080577) provides an image editing application for an interface that allows a user to easily change facial expression types, impression types, poses and rotation directions, etc. do.

Recently, various image conversion and synthesis methods using deep learning technology have emerged.

For example, Patent Document 2 (Japanese Patent Application Laid-Open No. 2019-148980) is a generative opposing neural network that is a deep learning-based generative model based on a non-camera gaze (the gaze does not match the camera) input image. (GAN: Generative Adversarial Networks) We propose a technology to convert the output image of the camera gaze.

To this end, Patent Document 2 requires a large number of sample facial images registered for learning in advance. In addition, to generate such a large number of sampled facial images, it is necessary to use a large number of cameras with known properties placed in multiple locations under controlled settings. Not only are these systems expensive and delicate, but they are also difficult to handle and not available to the average user. Of course, the identification accuracy of the face and the accuracy of the transformation vary, and it can benefit from many registered sample facial images, but providing these images greatly increases the burden on the user.

KR 2020-0080577 A JP 2019-148980 A

In this way, in the prior art, in converting a user's face into a learned face by applying a generative adversarial network (GAN), several pieces of image learning data with different angles and distances to create a desired facial image are used. had to collect That is, if various image learning data for the top, bottom, or side are not collected, an output image with severe cracking is inevitably generated when an input image of an angle and distance that does not correspond to an input image is received.

Accordingly, the technical problem to be achieved by the present invention is to provide a method of generating a plurality of image learning data having various angles and scales based on one sheet of image learning data.

Another technical task of the present invention is to provide an artificial intelligence deep learning model capable of generating a transformed image without breakage even at various angles and distances based on image learning data of one sheet.

In addition, another technical object of the present invention is to provide a method for converting a user's face into a learned face using a generative adversarial neural network (GAN).

GAN-based facial image conversion method according to the first aspect of the present invention for achieving the above-described technical problem, a plurality of face image learning data having various angles and distances by warping and scaling one source image A first step of creating a; a second step of generating a facial image conversion module by deep learning the facial image training data warped or scaled in the first step based on a generative alternative neural network (GAN); and a third step of generating an output image similar to the learned image by inputting an input image to be converted into the facial image conversion module.

A facial image conversion apparatus for converting an input image into an output image similar to a learned image according to another second aspect of the present invention includes: a memory for storing one or more instructions; and a processor executing the one or more instructions stored in the memory, wherein the processor generates a plurality of face image learning data having various angles and distances by warping and scaling one source image Step 1, the second step of generating a facial image conversion module by deep learning the facial image learning data based on a generative alternative neural network (GAN), and input the input image to be converted into the facial image conversion module and executing a process comprising a third step of generating an output image similar to the source image.

The first step may include: a warping step of generating a plurality of warped images rotated at a constant angle up, down, left, and right based on the one source image; and a scale conversion step of adjusting the scale of the facial image of the warping image.

The warping image is a rotation of the source image within a range of 20 degrees up and down and 30 degrees left and right, and the scale is adjusted from 0.25 times to 1.5 times.

In the warping step, the Affine transform network by generating several heatmaps from the source image, passing the heatmaps through the FC layer to obtain coefficients of the Affine transform equation, and deep learning the image sequentially based on the Affine transform equation , and generating the warping image through this Affine transform network.

The second step includes: converting the warped, scale-converted face image learning data into landmark learning data; calculating an arithmetic operation product of the style-encoded data of the facial image learning data; By taking the landmark learning data as the initial input of the generator of the GAN module, and inputting the arithmetic operation product for each network layer of the generator to make the identifier of the GAN module discriminate, the generating model and the discriminating model compete with each other and deep learning by and generating the facial image conversion module.

The step of calculating the calculation product is a matrix arithmetic operation product (Element-wise multiplication) of a sigmoid function of the feature obtained through the style encoding and the feature that the facial image learning data has passed through the layer of the network characterized in that

The third step may include: generating a deformation landmark of the input image to be converted; and generating, by the facial image conversion module, an output image similar to the learned image based on the deformation landmark.

The modified landmark is characterized in that the average and variance of the location of the landmark of the learned image are obtained, and the mean and variance of the location of the landmark of the input image are adjusted based on the mean and variance of the location.

An apparatus for training a facial image transformation operation according to another third aspect of the present invention comprises: one or more processors; and a memory coupled to the one or more processors comprising one or more program modules executable by the one or more processors to convert an input image to an output image similar to a learned image; The program module includes an image pre-processing module that rotates a source image at a certain angle up, down, left, and right, and adjusts the scale of the image to generate a plurality of face image learning data with different angles and distances from one source image and ; It includes a generative adversarial neural network (GAN)-based GAN module for deep learning while a generative model and a discrimination model compete for the plurality of facial image learning data.

The image preprocessing module includes: a warping unit for generating a plurality of warped images rotated at a predetermined angle up, down, left, and right based on the one source image; and a scale converter for adjusting the scale of the facial image of the warping image.

The warping unit generates a plurality of heatmaps from the source image, obtains the coefficients of the Affine transform formula by passing the heat map through the FC layer, and sequentially deep-leaves the image based on the Affine transform formula. It is a network.

The GAN module includes a generator and an identifier, and uses landmark training data generated from the warping and scale-converted facial image training data as an initial input to the generator, and arithmetic of the facial image training data style-encoded data It is a generative opposing neural network in which a generative model and a discriminant model compete in deep learning by inputting an operation product for each network layer of the generator and allowing the discriminator to discriminate.

The arithmetic operation product is a matrix arithmetic operation multiplication (Element-wise multiplication) of the sigmoid function of the feature obtained through the style encoding and the feature that the facial image learning data has passed through the network layer, characterized in that do.

According to the present invention, it is possible to generate multiple pieces of image learning data with different angles and distances from a single source image, and deep learning the image learning data to generate a face image transformation model based on a generative adversarial neural network. For this reason, since there is no need to collect multiple pieces of image learning data with different angles and distances, it is possible to provide an artificial intelligence-based image conversion technology that is inexpensive and delicate, and can reduce the user's burden. In addition, since an output image similar to an image learned from an input image having different angles and distances can be generated without breakage or distortion, it is possible to increase the accuracy of conversion for a face image.

Therefore, when the facial image conversion device of the present invention is used for video calls through character conversion or applied to 3D or VR games, it is possible to increase interest in video calls or immersion in games.

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and serve to further understand the technical spirit of the present invention together with the detailed description of the present invention to be described later, so that the present invention is a matter described in such drawings should not be construed as being limited only to
1 is a block diagram of a facial image conversion apparatus 100 according to a preferred embodiment of the present invention.
2 is a functional block diagram for explaining a GAN-based facial image conversion operation according to an embodiment of the present invention.
3 is a flowchart of a process of learning a face rotation generation model.
4 is a view showing facial images rotated in the range of 30 degrees left and right by the facial rotation generation model.
5 is a view showing facial images scaled from 0.25 times to 1.5 times by the scale converter.
6 is a diagram illustrating a learning structure of a generative alternative neural network (GAN) according to the present invention.
7 is a flowchart of an image conversion process using a regressed landmark according to the present invention.

Hereinafter, embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The embodiments of the present invention may be modified in various forms, and the scope of the present invention should not be construed as being limited to the following examples. This example is provided to more fully explain the present invention to those of ordinary skill in the art. In addition, although specific terms have been used in the drawings and the specification of the present invention, these are used only for the purpose of describing the present invention and are not used to limit the meaning or the scope of the present invention described in the claims. Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The accompanying drawings are not drawn to scale, and like reference numbers in each drawing refer to like elements.

1 is a block diagram of a facial image conversion apparatus 100 according to a preferred embodiment of the present invention.

The facial image converter 100 of the present invention may be part of a multifunctional device such as a mobile phone, a tablet computer, a personal digital assistant, a portable music/video player, a wearable device, or other electronic device including a camera system. In addition, the facial image converting apparatus 100 may be connected to other electronic devices through a network such as a network storage device such as a mobile device, a tablet device, a desktop device, and a server as an independent system. In addition, the facial image conversion device 100 may be connected to another electronic device through a wireless or wired connection.

The facial image conversion apparatus 100 may include a processor 110 . Processor 110 may be a system-on-a-chip, such as found in mobile devices, and may include one or more central processing units (CPUs), dedicated graphics processing units (GPUs), or both. Also, the processor 110 may include multiple processors of the same or different types.

The facial image conversion apparatus 100 may include a memory 150 . Memory 150 may include one or more different types of memory that may be used in conjunction with processor 110 to perform device functions. For example, memory 150 may include any type of non-transitory storage. The memory 150 may store various programming modules including the image preprocessing module 160 and the GAN module 170 during execution. The image pre-processing module 160 and the GAN module 170 may be stored in a memory other than the memory 150 including the memory of another electronic device. The image preprocessing module 160 and the GAN module 170 may include separate executable programming modules in some embodiments, but the functions of the programming modules may be combined into a single programming module.

The image pre-processing module 160 includes a warping unit 162, a scale conversion unit 164, a landmark unit 166, and a deformable landmark unit 168, as shown in FIG. 2, and the GAN module 170 is a Generative Adversarial Networks (GAN) including a generator 172 and an identifier 174 . The image pre-processing module 160 generates a plurality of face image learning data having a perspective of various angles by rotating a single source image up, down, left, and right to warp or adjust the scale. The GAN module 170 completes the facial image conversion module 170', which is a GAN module that has been learned by deep learning (deep learning) of the facial image learning data.

Also, the facial image conversion apparatus 100 may include one or more cameras 120 . Camera 120 may include image sensors, lens stacks, and other components that may be used to capture images. For example, camera 120 may be configured to capture an image of a particular face.

In one or more embodiments, the facial image converter 100 may include an input/output (I/O) device 140 . This I/O device 140 may be any kind of I/O device, such as a microphone for voice control input, or a speaker for audio data. For example, it can be a camera for visual data input, a display for visual data output (eg, any kind of display such as LCD, LED, organic LED (OLED), etc.)

Although facial image converter 100 in accordance with the present invention is illustrated as including a number of the components described above, in one or more embodiments, the various components are distributed across multiple devices as part of a distributed system. Additionally, additional components are available and some of the functions of the components can be combined.

2 is a functional flowchart for explaining a GAN-based facial image conversion operation according to an embodiment of the present invention. The facial image conversion operation is largely an image preprocessing process for generating learning data for training the GAN module 170, and a GAN learning process for generating a facial image conversion module 170' through deep learning of the GAN module 170 and It consists of a process of converting the input image 135 into an output image 180 similar to the learned source image through the facial image conversion module 170'.

First, the image pre-processing process is an operation programmed by the image pre-processing module 160 to warp or scale a source image to generate a plurality of image learning data having various angles and distances.

First, the warping unit 162 of the image preprocessing module 160 is rotated within the range of 30 degrees left and right and within the range of 20 degrees up and down as shown in FIG. 4 based on one source image 130. It is a deep learning-based facial rotation generation model that generates warping images of various angles.

3 is a flowchart of a process of learning a face rotation generation model. Referring to FIG. 3 , first, the source image 130 is divided by 24 fps. The T+1 second image 162a is passed through the convolutional network 162b to obtain 20 heatmaps 162c. The 20 heatmaps 162c pass through the FC layer (Fully-Connected Layer) 162d to obtain coefficients 162e of the Affine transform equation. Then, by multiplying the Affine transformation equation by the pixels (Pxi, Pyi, i=1 to N, N number of pixels) of the image for T seconds, respectively, a transformation prediction value 162f at T+1 seconds can be obtained. 3, 162g indicates an image frame of T seconds, and 162h indicates a predicted image frame obtained by multiplying the T-second image frame 162g and the Affine transform equation. By sequentially learning the images in this way, it is possible to obtain a face rotation generation model, which is a trained Affine transformation network.

The learned face rotation generation model generates a plurality of face image training data rotated within the range of 30 degrees to the left and 30 degrees to the right as shown in FIG. 4 from one source image 130 . According to the same principle, the learned face rotation generation model may also generate a plurality of face images rotated within the range of 20 degrees upward and 20 degrees downward.

When the rotation angle of the face image learning data exceeds the range of 30 degrees left and right and the range of 20 degrees up and down, the image may be severely distorted.

In this way, the face image learning data rotated at various angles by the warping unit 162 is scaled by the scale conversion unit 164 . That is, the scale conversion unit 164 can adjust the scale of the facial image learning data rotated at various angles from 0.25 times to 1.5 times as shown in FIG. 5 . In this case, the scale converter 164 fixes the size of the image as shown in FIG. 5 , and adjusts only the size of the face to give perspective to the face image.

In this way, the facial image learning data (I _o ) whose rotation angle and the facial scale are adjusted by the warping unit 162 and the scale converting unit 164 is converted into the landmark learning data (I _L ) by the landmark unit 166 . do. Here, the landmark indicates a feature point representing each feature of the object within the face detection area, for example, eyes, nose, mouth, and a feature point corresponding to an eyebrow or the like. At least one landmark may be set for each feature of the object, or a plurality of landmarks may be set for one feature. In this way, the number of features set for the object in the face detection area and the number of landmarks for each feature may be designated as needed.

Next, the _GAN learning process is a facial _image conversion module ( 170') is created.

The GAN module 170 is a generative adversarial network comprising a generator 172 and a discriminator 174 .

6 is a diagram illustrating a learning structure of a generative alternative neural network (GAN) according to the present invention.

Referring to FIG. 6 , the GAN module 170 takes the landmark learning data I _L as an initial input of the generator 172, and the facial image learning data I _o with the rotation angle and facial scale adjusted. By inputting element-wise multiplication of the style-encoded data for each network layer of the deep neural network of the generator 172 to generate a generated image, and having the identifier 174 discriminate it, the generative model and the discrimination model In this competition, deep learning completes the facial image conversion module 170', which is a GAN module that has been learned.

In more detail, the sigmoid function [M(F)] of the feature (F) obtained through style encoding for each layer of the deep neural network and the facial image training data (I _o ) are combined with the layers of the deep neural network. By performing matrix arithmetic operation (Element-wise multiplication) on the passed feature (F') and passing it through the layer of the deep neural network of the generator 172 together with the landmark learning data (I _L ), by the identifier 174 Generate image data to be discriminated.

In this case, when deep learning is performed by combining the facial image learning data (I _o ) and landmark learning data (I _L ) without style encoding, the image data may be severely distorted. Accordingly, the generator 172 of the GAN module 170 of the present invention can compensate for distortion of image data by taking only a portion with distinct features from the facial image through style encoding.

Finally, in the image conversion process, the input image 135 to be converted is input to the facial image conversion module 170 ', which is the learned GAN module, and the learned source image 130 is similar to the output image 180. It is a process of transformation.

At this time, if the landmark 136 of the input image 135 is directly input to the facial image conversion module 170 ′, the desired output image 180 cannot be generated because it is different from the learned image. Therefore, the input image 135 is generated as a landmark 137 deformed through the deformable landmark unit 168 of the image pre-processing module 160, and this deformed landmark 137 is converted into the facial image conversion module 170 ') is entered.

7 is a flowchart of an image conversion process using the deformable landmark 137 according to the present invention.

Referring to FIG. 7 , the deformation landmark 137 obtains the position average and variance of the landmarks (eyes, eyebrows, nose, mouth, and jaw line) of the learned facial image 145, and based on the position average and variance Thus, the average and variance of the positions of the landmarks 136 of the input image 135 are adjusted.

In this way, by inputting the modified landmark 137 for the input image 135 into the facial image conversion module 170 ′, it is possible to generate an output image 180 similar to a desired source image.

The operating method of the facial image conversion apparatus 100 according to an embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

In addition, the facial image conversion apparatus or the facial image conversion operation according to the disclosed embodiments may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities.

The computer program product may include a S/W program and a computer-readable storage medium in which the S/W program is stored. For example, computer program products may include products (eg, downloadable apps) in the form of S/W programs distributed electronically through manufacturers of electronic devices or electronic markets (eg, Google Play Store, App Store). there is. For electronic distribution, at least a portion of the S/W program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server temporarily storing a SW program.

The computer program product, in a system consisting of a server and a client device, may include a storage medium of the server or a storage medium of the client device. Alternatively, when there is a third device (eg, a smart phone) that is communicatively connected to the server or client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S/W program itself transmitted from the server to the client device or a third party device, or transmitted from the third device to the client device. In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of a server, a client device, and a third device may execute a computer program product to distribute the method according to the disclosed embodiments.

For example, a server (eg, a cloud server or an artificial intelligence server) may execute a computer program product stored in the server to control a client device communicatively connected with the server to perform the method according to the disclosed embodiments.

In the above, although the present invention has been described with reference to limited embodiments and drawings, the present invention is not limited thereto and will be described below with the technical idea of the present invention by those of ordinary skill in the art to which the present invention pertains. Of course, various modifications and variations are possible within the scope of equivalents of the claims.

100: facial image converter, 110: processor, 120: camera, 130: source image, 135: input image, 136: landmark, 137: deformed landmark, 140: I/O device, 150: memory, 160: image Pre-processing module, 162: warping unit, 164: scale conversion unit, 166: landmark unit, 170: GAN module, 170': facial image conversion module, 172: generator, 174: identifier, 180: output image

Claims (21)

A first step of generating a plurality of face image training data having various angles and distances by warping and scaling one source image;
a second step of generating a facial image conversion module by deep learning the facial image learning data based on a generative adversarial neural network (GAN); and
GAN-based facial image conversion method comprising a third step of generating an output image similar to the learned image by inputting an input image to be converted into the facial image conversion module. According to claim 1, wherein the first step,
a warping step of generating a plurality of warped images rotated at a constant angle up, down, left, and right based on the one source image;
GAN-based facial image conversion method comprising a; scale conversion step of adjusting the scale of the facial image of the warping image. 3. The method of claim 2,
The warping image is a GAN-based facial image conversion method, characterized in that the source image is rotated within a range of 20 degrees up and down, and within a range of 30 degrees left and right. According to claim 3, wherein the warping step,
Affine transformation network is obtained by generating several heatmaps from the source image, obtaining coefficients of the Affine transformation equation by passing the heatmap through the FC layer, and deep learning the image sequentially based on the Affine transformation equation. GAN-based facial image conversion method, characterized in that generating the warping image through a transformation network. 3. The method of claim 2,
GAN-based facial image conversion method, characterized in that the scale is adjusted from 0.25 times to 1.5 times. According to claim 1, wherein the second step,
converting the warping and scale-converted facial image learning data into landmark learning data;
calculating an arithmetic operation product of the style-encoded data of the facial image learning data;
By taking the landmark learning data as the initial input of the generator of the GAN module, and inputting the arithmetic operation product for each network layer of the generator to make the identifier of the GAN module discriminate, the generating model and the discriminating model compete with each other and deep learning by GAN-based facial image conversion method comprising the step of generating the facial image conversion module. The method of claim 6, wherein the calculating of the product of the calculation operation comprises:
GAN-based facial, characterized in that the sigmoid function of the feature output through the style encoding and the feature that the facial image learning data passes through the network layer are multiplied by matrix arithmetic operation (Element-wise multiplication) How to convert an image. According to claim 1, wherein the third step,
generating a deformation landmark of the input image to be converted;
GAN-based facial image conversion method comprising the step of generating, by the facial image conversion module, an output image similar to the learned image based on the deformation landmark. 9. The method of claim 8,
The modified landmark obtains the position average and variance of the landmarks of the learned image, and based on the position average and variance, the average and variance of the positions of the landmarks of the input image are adjusted based on the GAN-based How to convert facial image. A device for training facial image transformation behavior, comprising:
one or more processors; and
a memory coupled to the one or more processors comprising one or more program modules executable by the one or more processors to convert an input image into an output image similar to a learned image;
The program module is
an image pre-processing module for generating a plurality of face image learning data with different angles and distances from one source image by rotating the source image at a constant angle up, down, left, and right and adjusting the scale of the image;
An apparatus for training a facial image transformation operation, characterized in that it includes a generative adversarial neural network (GAN)-based GAN module for deep learning while a generative model and a discrimination model compete for the plurality of facial image learning data. 11. The method of claim 10, wherein the image pre-processing module,
a warping unit generating a plurality of warping images rotated at a predetermined angle up, down, left, and right based on the one source image;
An apparatus for training a facial image transformation operation comprising a; a scale transformation unit for adjusting the scale of the facial image of the warping image. The method of claim 11, wherein the warping unit,
It is an Affine transformation network generated by generating several heatmaps from the source image, obtaining coefficients of the Affine transformation equation through the FC layer, and deep learning the image sequentially based on the Affine transformation equation A device that trains facial image transformation behavior with 12. The method of claim 11,
The GAN module includes a generator and an identifier,
The landmark learning data generated from the warping and scale-converted facial image learning data is an initial input to the generator, and the arithmetic operation product of the style-encoded data of the facial image learning data is input for each network layer of the generator. A device for training facial image transformation behavior, characterized in that it is a generative adversarial neural network that deep learns while the generative model and the discriminant model compete by making the discriminator discriminate. 14. The method of claim 13,
The arithmetic operation product is a matrix arithmetic operation multiplication (Element-wise multiplication) of the sigmoid function of the feature obtained through the style encoding and the feature that the facial image learning data has passed through the network layer, characterized in that A device that trains facial image transformation movements. A facial image converter that converts an input image into an output image similar to a learned image,
a memory storing one or more instructions; and
a processor executing the one or more instructions stored in the memory;
The processor is
A first step of generating a plurality of face image training data having various angles and distances by warping and scaling one source image,
A second step of generating a facial image conversion module by deep learning the facial image learning data based on a generative alternative neural network (GAN), and
Facial image conversion device, characterized in that executing a process comprising a third step of generating an output image similar to the learned source image by inputting an input image to be converted into the face image conversion module. The method of claim 15, wherein the first step comprises:
a warping step of generating a plurality of warped images rotated at a constant angle up, down, left, and right based on the one source image;
Scale conversion step of adjusting the scale of the facial image of the warping image; Facial image conversion device comprising a. The method of claim 16, wherein the warping step comprises:
Affine transformation network is obtained by generating several heatmaps from the source image, obtaining coefficients of the Affine transformation equation by passing the heatmap through the FC layer, and deep learning the image sequentially based on the Affine transformation equation. Facial image conversion device, characterized in that for generating the warping image through a conversion type network. The method of claim 15, wherein the second step comprises:
converting the warping and scale-converted facial image learning data into landmark learning data;
calculating an arithmetic operation product of the style-encoded data of the facial image learning data;
By taking the landmark learning data as the initial input of the generator of the GAN module, and inputting the arithmetic operation product for each network layer of the generator to make the identifier of the GAN module discriminate, the generating model and the discriminating model compete with each other and deep learning by Facial image conversion device comprising the step of generating the face image conversion module. 19. The method of claim 18, wherein the step of calculating the calculation operation product,
Facial image conversion apparatus, characterized in that the sigmoid function of the feature output through the style encoding and the feature that the facial image learning data passes through the network layer are multiplied by matrix arithmetic operation (Element-wise multiplication) . 16. The method of claim 15, wherein the third step comprises:
generating a deformation landmark of the input image to be converted;
Facial image conversion device, characterized in that it comprises the step of generating, by the facial image conversion module, an output image similar to the learned image based on the deformation landmark. 21. The method of claim 20,
The modified landmark obtains the position average and variance of the landmarks of the learned image, and based on the position average and variance, adjusts the average and variance of the positions of the landmarks of the input image. Device.

Images