KR102590025B1 - Learning method of face swapping deep learning system that increases learning efficiency through attention mask - Google Patents

Abstract

The present invention relates to a method of learning a face swapping deep learning system, including (a) inputting a source image 100 into an embedding network 10 to extract a first feature embedding vector 110; (b) Converting pixels of the target image 200 to a compatible size and converting them into target data 210; (c) combining the target data 210 and the first feature embedding vector 110 to perform a first composite transformation Generating data 300; (c') combining the target data 210 and the first synthesized converted data 300 with an attention mask 310 to generate second synthesized converted data 320. Step;, (d) a composite image conversion step of converting the second composite conversion data 320 into a swapping image 400; and (e) learning the face swapping deep learning system based on the swapping image 400.

Description

Learning method of face swapping deep learning system that increases learning efficiency through attention mask}

The present invention relates to a learning method for a face swapping deep learning system that increases learning efficiency through an attention mask. More specifically, it is about a method of learning a system that synthesizes the identity and attributes of two people's images by differentiating the degree of learning.

Patent Document 001 discloses a method and device for synthesizing images by executing an artificial intelligence (AI) algorithm and/or a machine learning algorithm in a 5G environment connected for the Internet of Things. An image synthesis method according to an embodiment of the present invention includes acquiring a first image including a face image, detecting feature points of the first image by applying a pre-trained deep neural network model, and Obtaining a second image for compositing, extracting the boundary of the second image, matching coordinate values corresponding to the boundary of the second image based on the coordinate values of the feature points of the first image, A technology including the step of merging a first image and a second image and outputting them is presented.

Patent Document 002 is about a method for generating images such as facial expressions based on AI machine learning, and the second server receives the facial expression information of each actor from the first server, performs AI image learning on the file, and performs AI image learning on the file. As a result of AI image learning, a weight corresponding to the actor's facial expression is created and stored, a second server receives the actor's facial expression information from the first server, and the second server receives the actor's facial expression. We present a technology that includes the steps of searching for the weight corresponding to the actor, generating and storing the facial expression of the actor, and generating a video (VOD file, etc.) with the facial expression of the actor on the second server. .

Patent Document 003 is an AI-based character creation method that is performed by an AI-based character creation system, including the step of learning a character creation model using learning data including phenotypic genome information and character image data; and directly or indirectly from the user. It includes the step of generating a character image by applying the character creation model to the phenotypic genome information provided. Because each character has their own genomic information, we present a technology that makes it possible to give uniqueness and identification to characters in virtual space.

Patent Document 004 relates to a convolutional neural network-based image processing system, which includes a convolutional layer unit that extracts feature values of an object from input data using a plurality of kernel filters; an activation layer unit that performs a transformation operation on the feature values extracted by the convolution layer unit using a non-linear activation function; a pooling layer unit that reduces the dimensionality of the output value of the activation layer unit using a max polling operation and removes and suppresses noise; a classification output layer unit that outputs a classification prediction value for the input data through a forward operation using the output value of the pooling layer unit; a loss calculation layer unit that compares the classification prediction value with a predetermined target value and calculates a loss value corresponding to the error value; a reverse calculation unit that calculates a partial differential value for the loss value through a reverse operation to obtain correction values for the parameters of each layer; and a classification learning unit that performs an update on the parameter through a gradient descent method using the correction value and a learning rate derived from a certain amount of learning data.

KR 10-2236904 (Registration date: March 31, 2021) KR 10-2021-0112576 (Publication date: September 15, 2021) KR 10-2022-0155239 (Publication date: November 22, 2022) KR 10-2068576 (Registration date: January 15, 2020)

The present invention seeks to provide a method and device for learning a system for converting multiple face images into one composite image.

In addition, we would like to provide a swapping image synthesized by classifying the characteristics of each input face image and then combining only the desired characteristics.

The invention according to an embodiment of the present invention relates to a method of learning a face swapping deep learning system, comprising: (a) inputting a source image into an embedding network to extract a first feature embedding vector; (b) target image converting to a compatible size and converting it into target data; (c) combining the target data and the first characteristic embedding vector to generate first synthesized converted data; (d) the first synthesized converted data A composite image conversion step of converting into a swapping image; and (e) learning the face swapping deep learning system based on the swapping image.

The invention according to an embodiment of the present invention relates to a method of learning a face swapping deep learning system. In the invention presented above, step (c) converts the target data (X) into target regular data (Z). Normalization step to convert; and a denormalization step of combining the target normal data (Z) and the identity data extracted from the first feature embedding vector (110).

The invention according to an embodiment of the present invention relates to a method of learning a face swapping deep learning system, and in the invention presented above, the normalization step is calculated by the following [Equation 1].

[Equation 1]

(step, class means the average and standard deviation of target data (X))

The invention according to an embodiment of the present invention relates to a method of learning a face swapping deep learning system, and in the invention presented above, the denormalization step is calculated by the following [Equation 2].

[Equation 2]

(step, means identity data)

The invention according to an embodiment of the present invention relates to a method of learning a face swapping deep learning system. In the invention presented above, step (e) is (e-1) inputting the swapping image to the embedding network. After extracting the second feature embedding vector, learning the face swapping deep learning system in a way to reduce the first error, which is the difference between the first feature embedding vector and the second feature embedding vector. It comes true.

The invention according to an embodiment of the present invention relates to a method of learning a face swapping deep learning system. In the invention presented above, step (e) is (e-2) by inputting the swapping image into a discriminator network. Extracting first intermediate values in a calculation process; (e-3) inputting the target image into a discriminator network to extract second intermediate values in a calculation process; and (e-4) learning the face swapping deep learning system in a direction to reduce a second error, which is the difference between the first median and the second median.

The invention according to an embodiment of the present invention relates to a method of learning a face swapping deep learning system, comprising: (a) inputting a source image into an embedding network to extract a first feature embedding vector; (a-1) Input the source image into a 3DMM (3D Morphable Model) network to extract a source 3D embedding vector, (a-2) Input the target image into a 3DMM (3D Morphable Model) network to extract a target 3D embedding vector, (a-2) a-3) Generate a synthetic 3D embedding vector by combining the source 3D embedding vector and the target 3D embedding vector, and (a-4) Connect the synthetic 3D embedding vector to the first feature embedding vector to create a 3D feature embedding vector. (b) converting pixels of the target image to a compatible size and converting them into target data; (c) generating second composite converted data by combining the target data and the 3D feature embedding vector; (d) composite image conversion step of converting the second composite converted data into a swapping image; (e) It consists of learning the face swapping deep learning system based on the swapping image.

The invention according to an embodiment of the present invention relates to a method of learning a face swapping deep learning system, comprising: (a) inputting a source image into an embedding network to extract a first feature embedding vector; (b) target image Converting pixels to a compatible size and converting them into target data; (c) combining the target data and the first feature embedding vector to generate first synthesized converted data; (c') the target data and generating second synthesized converted data by combining the first synthesized converted data with an attention mask; (d) a synthesized image conversion step of converting the second synthesized converted data into a swapping image; (e) the swapping It consists of a configuration including; learning the face swapping deep learning system based on the image.

The invention according to an embodiment of the present invention relates to a method of learning a face swapping deep learning system. In the invention presented above, the step (c') is (c'-1) the attention mask is converted to a sigmoid. It consists of a configuration including a step of creating based on a network.

The invention according to an embodiment of the present invention relates to a computer-implemented learning device for learning a face swapping deep learning system, comprising: a memory for storing instructions; and at least one processor configured to execute the instructions, wherein the instructions executed through the processor input a source image into the embedding network 10 to extract a first feature embedding vector and convert the target image into a compatible size. Convert to target data, generate synthesized converted data by combining the target data and the first feature embedding vector, convert the synthesized converted data into a swapping image, and perform the face swapping dip based on the swapping image. It consists of a computer-implemented learning device that learns the learning system.

The present invention can provide a method for effectively learning a system that combines the attributes of a target image and the identity of a source image.

Additionally, the present invention can provide a method for effectively learning a configuration to be intensively learned through an attention mask.

1 is a flowchart of the operation of a face swapping deep learning system according to an embodiment of the present invention.
Figure 2 is a schematic diagram illustrating a process in which a source image is input to an embedding network and a first feature embedding vector is generated according to an embodiment of the present invention.
Figure 3 is a detailed configuration diagram expressing the calculation principle of a mix block according to an embodiment of the present invention.
Figure 4 is a block diagram showing a first error according to an embodiment of the present invention.
Figure 5 is a block diagram showing a second error according to an embodiment of the present invention.
Figure 6 is a schematic diagram showing the generation process of first synthetic conversion data and attention mask according to an embodiment of the present invention.
Figure 7 is a schematic diagram showing the process of generating second synthetic converted data by applying an attention mask according to an embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, but may be implemented in various different forms, and should be understood to include all conversions, equivalents, and substitutes included in the spirit and technical scope of the present invention. .

The embodiments presented below are provided to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. In describing the present invention, if it is determined that a detailed description of related known technologies may obscure the gist of the present invention, the detailed description will be omitted.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, identical or corresponding components are assigned the same drawing numbers and duplicate descriptions thereof are omitted. I decided to do it.

(Example 1-1) In a method of learning a face swapping deep learning system, (a) inputting a source image 100 into an embedding network 10 to extract a first feature embedding vector 110; (b) converting the target image 200 to a compatible size and converting it into target data 210; (c) combining the target data 210 and the first feature embedding vector 110 to perform a first synthesis Generating converted data 300; (d) Composite image conversion step of converting the first composite converted data 300 into a swapping image 400; (e) Based on the swapping image 400 A method of learning the face swapping deep learning system, including the step of learning the face swapping deep learning system.

An exemplary embodiment of the present invention may relate to a method of learning a face swapping deep learning system that synthesizes characteristics of multiple people.

Referring to FIGS. 1 to 7, the face swapping deep learning system according to an exemplary embodiment of the present invention includes a convolution block 30, a convolution layer 31, a linear layer 20, a mix block 40, etc. It can mean a combination of modules that convert and synthesize digital data. Therefore, learning the face swapping deep learning system in the present invention may mean learning the weights, nodes, etc. of various modules as described above.

The present invention includes the steps of inputting the source image 100 into the embedding network 10 to extract the first feature embedding vector 110, converting the pixels of the target image 200 to a compatible size to produce a first conversion target image 200. ), generating a composite transformation image by combining the first transformation target image 200 and the first feature embedding vector 110, and converting the synthesis transformation image into a swapping image 400. An image conversion step may be included.

In more detail, an exemplary embodiment of the present invention includes the identity of the source image 100 (shape information of a person's eyes, nose, etc.) and the attributes of the target image 200 (information such as the angle of a person's face, lighting, etc.) ) It may be about the learning method of a deep learning model that synthesizes.

Hereinafter, specific methods for each process will be described later.

(Example 1-2) In Example 1-1, the source image 100, the target image 200, and the swapping image 400 may be RGB 3-channel images.

In an exemplary embodiment of the present invention, the source image 100, target image 200, and swapping image 400 are It may be a pixel size of .

Currently, images expressed digitally in the industry may be images composed of RGB. This may be an image composed of three channels, and therefore the target image 200 and source image 100 used for face swapping, and the final synthesized swapping image 400 may be a three-channel image.

(Example 1-3) In Example 1-1, the compatible size is It may be a pixel size of .

In an exemplary embodiment of the present invention, the compatible size may be a size configured before the target image 200 is synthesized with the first feature embedding vector 110. Referring to Figure 1, You can set the pixel size to a compatible size. However, the compatible size is not specified, and the compatible size can be set in various ways if the process of combining the target data 210 converted to the compatible size and the first feature embedding vector 110 can be more efficient.

Additionally, in an exemplary embodiment of the present invention, the first feature embedding vector 110 may be a vector output when the source image 100 is input to the already sufficiently learned embedding network 10. The pixel size of this first feature embedding vector 110 is It can be set to , and this can be information consisting of 512 numbers.

Accordingly, the channel size of the target data 210 and the channel size of the first feature embedding vector 110 may be set to be the same. Two data with channels of the same size can be combined through a convolution process.

(Example 2-1) Step (c) is a normalization step of converting the target data 210 (X) into target normal data 220 (Z);, the target normal data 220 (Z) It includes a denormalization step of synthesizing the identity data 120 extracted from the first feature embedding vector 110.

In an exemplary embodiment of the present invention, the step of generating synthesized converted data by combining the target data 210 and the first feature embedding vector 110 may include a normalization step and a denormalization step. In the normalization step and the denormalization step, the target data 210 is configured in the form of a normal distribution, and the identity data 120 extracted from the first feature embedding vector 110 is added to the target normal data 220 configured in the form of this normal distribution. ) may refer to the step of generating first synthesized converted data 300 (X') by synthesizing.

(Example 2-2) In Example 2-1, the normalization step is a process calculated by the following [Equation 1].

[Equation 1]

(step, class means the average and standard deviation of target data (210))

(Example 2-3) In Example 2-2, the denormalization step is a process calculated by the following [Equation 2].

[Equation 2]

(step, means identity data (120))

(Example 2-4) In Example 2-3, the denormalization step includes extracting the identity data 120 by inputting the first feature embedding vector 110 into the linear layer 20. Includes.

In an exemplary embodiment of the present invention, the first feature embedding vector 110 may be data including a plurality of numbers. From this first feature embedding vector 110, the identity data 120 is After generating, the target regular data 220 can be converted into first synthetic conversion data 300 using the identity data 120.

In the process of learning the face swapping deep learning system of the present invention, the weight values that make up the calculation algorithm of the linear layer 20 can be learned.

To explain more specifically,

The first feature embedding vector 110 is If it is a vector consisting of 512 numbers It may be a configuration expressed as . Through these 512 numbers, the identity data (120) is as follows: can be created.

Therefore, the identity data 120 from the first feature embedding vector 110 How to extract is the weight of the linear layer (20). It can be determined according to the values. The value of the first synthetic conversion data 300 generated in step (c) of the present invention can be determined depending on how these weights are configured. Therefore, the method of learning the face swapping deep learning system of the present invention may include learning these weights.

(Example 3-1) In Example 1-1, step (e) was

(e-1) After inputting the swapping image 400 into the embedding network 10 and extracting the second feature embedding vector 410, the first feature embedding vector 110 and the second feature embedding vector ( 410), learning the face swapping deep learning system in the direction of reducing the first error (1), which is the difference.

In an exemplary embodiment of the present invention, the first error 1 may refer to an error related to identity. If sufficiently learned, the identity of the swapping image 400, which is a composite of the source image 100 and the target image 200, is the same as the identity of the source image 100, and the attribute of the swapping image 400 is the target image 200. It can be configured in the same way as . If the above results are obtained, the face swapping deep learning system of the present invention can be considered to have been learned most effectively.

In <Example 3-1>, learning the system of the present invention in the direction of reducing the first error (1) means that when the face swapping deep learning system synthesizes the source image 100 and the target image 200, the source image 100 and the target image 200 are combined. This may mean learning so that the identity value of the image 100 can be maintained in the swapping image 400, which is a synthetic image.

(Example 4-1) In Example 3-1, step (e) was

(e-2) inputting the swapping image 400 into the discrimination network 50 and extracting first intermediate values 51 during the calculation process;

(e-3) inputting the target image 200 into the discrimination network 50 and extracting second intermediate values 52 during the calculation process;

(e-4) learning the face swapping deep learning system in a direction to reduce a second error (2), which is the difference between the first median value (51) and the second median value (52);

Includes.

In an exemplary embodiment of the present invention, the second error 2 may be configured to include a loss related to an attribute.

In <Example 4-1>, learning the system of the present invention in the direction of reducing the second error 2 means that when the face swapping deep learning system synthesizes the source image 100 and the target image 200, the target This may include learning to maintain the attribute values of the image 200 in the swapping image 400, which is a synthetic image.

In addition, referring to <Example 4-1> of the present invention, when the face swapping deep learning system is trained in the direction of reducing the second error 2, information about the identity of the target image 200 is stored in the swapping image. There may be a possibility that learning occurs in the direction reflected in (400). However, the degree to which the face swapping deep learning system is learned in the direction of reducing the first error (1) is more predominantly reflected, and the information about the identity is not the information contained in the target image 200, but the source image ( 100) can be learned to reflect the information contained in it.

(Example 4-2) In Example 4-1, the discrimination network 50 is composed of a discriminator.

The discriminator may be a network that determines whether an input image is an image of a person or not. The discrimination network 50 may be a network that outputs a number between 0 and 1 for an input image. If it is judged to be true, the number 1 is output, and if it is judged to be false, the number 0 is output.

The discrimination network 50 may produce intermediate values in the process of outputting numbers from 0 to 1, which are final output values. The intermediate values of various stages may be data values that are continuously generated in the process of converting the input image to values of 0 to 1.

When the target image 200 is input to the discrimination network 50, the final output value may be close to 1 because the target image 200 is originally an image of a person.

However, since the swapping image 400 is a composite image of the target image 200 and the source image 100, it may not appear human-like until the face swapping deep learning system of the present invention is sufficiently trained. An output value close to 0 can be derived.

However, the discrimination network 50 of the present invention may not be configured to derive an output value of 0 to 1. The face swapping deep learning system of the present invention may simply be learned using intermediate values produced in the process of generating output values.

(Example 5-1) In Example 1-1, in the method of learning a face swapping deep learning system,

(a) inputting the source image 100 into the embedding network 10 to extract a first feature embedding vector 110;

(a-1) Input the source image 100 into a 3DMM (3D Morphable Model) network to extract the source 3D embedding vector,

(a-2) Input the target image 200 into a 3DMM (3D Morphable Model) network to extract a target 3D embedding vector,

(a-3) Generating a synthetic 3D embedding vector by combining the source 3D embedding vector and the target 3D embedding vector,

(a-4) generating a 3D feature embedding vector by connecting the synthesized 3D embedding vector to the first feature embedding vector 110;

(b) converting pixels of the target image 200 to a compatible size and converting them into target data 210;

(c) generating second synthesized converted data 320 by combining the target data 210 and the 3D feature embedding vector 110;

(d) a composite image conversion step of converting the second composite conversion data 320 into a swapping image 400;

(e) A method of learning the face swapping deep learning system, including the step of learning the face swapping deep learning system based on the composite image 400.

In an exemplary embodiment of the present invention, when a source image or target image is input to a 3DMM network, a 3D embedding vector including 3D information of the image may be output. The 3D embedding vector may be a vector consisting of 257 numbers.

3DMM may refer to a deformable 3D movement model, and may refer to a model that generates movement and/or facial expressions by applying various techniques to a 3D model. Here, the deformable 3D motion model may refer to a 3D motion model included in an animation to which the Morphable 3D model technique is applied. For example, in a deformable 3D motion model, 3D shape and/or texture transformations within an object may be continuously parameterized, establishing a mapping between the lower-dimensional parameter space and/or the higher-dimensional space of the textured 3D model. there is.

However, in the present invention, a 3DMM network may mean a network that extracts a one-dimensional 3D embedding vector based on a 3DMM model when an image is input.

Therefore, the source image can be input to the 3DMM network to extract the source 3D embedding vector, the target image can be input to the 3DMM network to extract the target 3D embedding vector, and then the two 3D embedding vectors can be combined to generate a synthetic 3D embedding vector. .

Information related to the identity of the source image can be extracted from the source 3D embedding vector, and information related to the attributes of the target image can be extracted and synthesized from the target 3D embedding vector.

The 3D embedding vector extracted from the 3DMM network may be a vector extracted by a different algorithm from the feature embedding vectors extracted from the above-described embedding network 10.

In this way, the 3DMM network may be a network that extracts 3D information of images as described above. Furthermore, a 3D feature embedding vector can be created by connecting the synthesized 3D embedding vector and the first feature embedding vector 110, and the 3D feature embedding vector is used to synthesize the target data as in <Example 1-1>. 2 Synthetic conversion data 320 can be generated.

Finally, the swapping image 400, which is the final image, can be generated based on the second synthetic conversion data 320.

In other words, the present invention may be about a method of learning the face swapping deep learning system of the present invention by also reflecting the 3D characteristics of the image.

(Example 6-1) A method for learning a face swapping deep learning system, comprising: (a) inputting a source image 100 into an embedding network 10 to extract a first feature embedding vector 110;

(b) converting pixels of the target image 200 to a compatible size and converting them into target data 210;

(c) generating first synthesized converted data 300 by combining the target data 210 and the first feature embedding vector 110;

(c') combining the target data 210 and the first synthesized converted data 300 with an attention mask 310 to generate second synthesized converted data 320;

(d) a composite image conversion step of converting the second composite conversion data 320 into a swapping image 400;

(e) learning the face swapping deep learning system based on the swapping image 400;

Includes.

(Example 6-2) In Example 6-1, step (c') was

(c'-1) generating the attention mask 310 based on the sigmoid network 60.

In an exemplary embodiment of the present invention, the sigmoid network 60 may be a network that generates an output value of 0 to 1 based on a sigmoid function.

Sigmoid function inputs When entered, It is a function that outputs, The value of consists of values from 0 to 1 as shown in [Equation 3] below.

[Equation 3]

(Example 6-3) In Example 6-2,

(c'-2) assuming that the generated function of the attention mask 310 is M, the second synthesized converted data 320 is calculated using Equation 4 below;

[Equation 4]

(However, X is the target data 200,

Includes.

(Example 6-4) In Example 6-1, (e) the swapping image 400 is input to the embedding network 10 to extract the second feature embedding vector 410, and then the first feature It includes: learning the face swapping deep learning system in a direction to reduce the first error 1, which is the difference between the embedding vector 110 and the second feature embedding vector 410.

(Example 6-5) In Example 6-4, (g) inputting the swapping image 400 into the discrimination network 50 and extracting first intermediate values 51 during the calculation process;, ( h) inputting the target image 200 into the discrimination network 50 to extract second intermediate values 52 during the calculation process; (i) the first intermediate value 51 and the second intermediate value Learning the face swapping deep learning system in a direction to reduce the second error (2), which is the difference of (52);

Includes.

The sigmoid network 60 of the present invention may be a configuration learned during the learning process of the face swapping deep learning system of the present invention.

<Example 1-1> synthesizes the target data 210 generated from the target image 200 and the first feature embedding vector 110 generated from the source image 100 to produce first synthesized converted data 300. It includes the step of generating and using this to generate a swapping image 400 that combines the characteristics of the target image 200 and the source image 100.

In contrast, in Example 6-1, in compositing the target image 200 and the source image 100, the swapping image 400 maintains the attribute characteristics of the target image 200 and the source image 100. A face swapping deep learning system that maintains identity characteristics more efficiently can provide a learning method.

To ensure the above effect, an attention mask 310 can be additionally introduced.

The attention mask 310 may be a component that assists the process of generating the second synthesized converted data 320 by combining the target data 210 and the first synthesized converted data 300. When the function of the attention mask 310 is M,

[Equation 4]

It can be calculated through [Equation 4] as above. The part where the M value is close to 1 follows the first synthetic conversion data (300), and the part where the M value is close to 0 can follow the target data (210). there is.

In other words, the target data 210 and the first feature embedding vector 110 may be synthesized to generate first synthesized converted data 300. For the part where the characteristics of the target image 200 are to be maintained, that is, the part corresponding to the attribute information, the configuration value of the attention mask 310 is formed close to 0, and the characteristics of the source image 100 are to be maintained, That is, for the part corresponding to the identity information, the attention mask 310 configuration value can be learned to be close to 1.

The second synthetic conversion data 320 is generated using the attention mask 310, the swapping image 400 is generated using this, and then the first error 1 and the second error 2 are calculated using this. After extraction, the face swapping deep learning system of the present invention can be trained to reduce them.

Regarding the contents of Example 6-4 and Example 6-5, the contents of Example 3-1 and Example 4-1 described above can be applied mutatis mutandis.

However, it cannot be used if the second synthetic conversion data 320 is generated using the attention mask 310, the swapping image 400 is generated, and the first error (1) and the second error (2) and learning are performed. Learning efficiency can be higher than if it were not done.

This may be because the attention mask 310 can play a role in emphasizing the part to be focused on in each image when combining two images with respect to the characteristics of the target image 200 and the source image 100. .

(Example 7-1) A computer-implemented learning device for learning a face swapping deep learning system, comprising: a memory for storing instructions; and at least one processor configured to execute the instructions, wherein the instructions executed through the processor input the source image 100 into the embedding network 10 to extract the first feature embedding vector 110. , converting the target image 200 to a compatible size and converting it into target data 210, synthesizing the target data 210 and the first feature embedding vector 110 to generate synthetic conversion data, and generating the synthetic conversion. It is a computer-implemented learning device that converts data into a swapping image 400 and learns the face swapping deep learning system based on the swapping image 400.

(Example 7-2) In Example 7-1, the command inputs the swapping image 400 into the embedding network 10 to extract the second feature embedding vector 410, and then extracts the first feature embedding vector 410. A computer-implemented learning device that learns the face swapping deep learning system in a way to reduce the first error (1), which is the difference between the embedding vector (110) and the second feature embedding vector (410).

(Example 7-3) In Example 7-2, the command inputs the swapping image 400 to the discrimination network 50 to extract first intermediate values 51 in the calculation process, and the target image (200) is input into the discrimination network 50 to extract second median values 52 during the calculation process, and a second error ( 2) A computer-implemented learning device that learns the face swapping deep learning system in the direction of reducing.

An exemplary embodiment of the present invention relates to a computer-implemented learning device for learning a face swapping deep learning system, and may include a memory for storing instructions and at least one processor configured to execute the instructions. The processor may execute instructions to perform steps from (Embodiment 1-1) to (Embodiment 6-5). Overlapping content will be omitted.

In the specification (particularly in the claims) of the present invention, the use of the term “above” and similar referential terms may refer to both the singular and the plural. In addition, when a range is described in the present invention, the invention includes the application of individual values within the range (unless there is a statement to the contrary), and each individual value constituting the range is described in the detailed description of the invention. It's the same.

Unless there is an explicit order or statement to the contrary regarding the steps constituting the method according to the invention, the steps may be performed in any suitable order. The present invention is not necessarily limited by the order of description of the above steps. The use of any examples or illustrative terms (e.g., etc.) in the present invention is merely to describe the present invention in detail, and unless limited by the claims, the scope of the present invention is limited by the examples or illustrative terms. It doesn't work. Additionally, those skilled in the art will recognize that various modifications, combinations and changes may be made depending on design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all scopes equivalent to or equivalently changed from the scope of the claims are within the scope of the spirit of the present invention. It will be said to belong to

10: Embedding network 20: Linear layer
30: Convolution block 40: Mix block
50: Discriminant network 60: Sigmoid network
100: Source image 110: First feature embedding vector
120: Identity data 200: Target image
210: target data 220: target regular data
300: First synthetic conversion data 310: Attention mask
320: Second composite conversion data 400: Swapping image

Claims (6)

In a method of learning a face swapping deep learning system,
(a) inputting the source image 100 into the embedding network 10 to extract a first feature embedding vector 110;
(b) converting pixels of the target image 200 to a compatible size and converting them into target data 210;
(c) generating first synthesized converted data 300 by combining the target data 210 and the first feature embedding vector 110;
(c') combining the target data 210 and the first synthesized converted data 300 with an attention mask 310 to generate second synthesized converted data 320;
(d) a composite image conversion step of converting the second composite conversion data 320 into a swapping image 400; and
(e) learning the face swapping deep learning system based on the swapping image 400;
Including,
The step (c) includes a normalization step of converting the target data 210 (X) into target normal data 220 (Z); and
A denormalization step of combining the target normal data 220 (Z) and the identity data 120 extracted from the first feature embedding vector 110,
The normalization step is calculated using Equation 1 below,
[Equation 1]

(step, class means the average and standard deviation of target data (210))

The denormalization step inputs the first feature embedding vector 110 into the linear layer 20 to extract the identity data 120, and is calculated based on the following [Equation 2],
[Equation 2]

(step, means identity data (120)

The step (e) is
(e-1) After inputting the swapping image 400 into the embedding network 10 and extracting the second feature embedding vector 410, the first feature embedding vector 110 and the second feature embedding vector ( Learning the face swapping deep learning system in a direction to reduce the first error (1), which is the difference 410);
(e-2) inputting the swapping image 400 into the discrimination network 50 and extracting first intermediate values 51 during the calculation process;
(e-3) inputting the target image 200 into the discrimination network 50 and extracting second intermediate values 52 during the calculation process; and
(e-4) learning the face swapping deep learning system in a direction to reduce the second error (2), which is the difference between the first median value (51) and the second median value (52); ,
The first feature embedding vector 110 is 512 numbers. It is data composed of, and the identity data 120 is generated as follows:


How to train a face swapping deep learning system.
In claim 1,
The step (c') is
(c'-1) generating the attention mask 310 based on a sigmoid network 60; a method of learning a face swapping deep learning system, including.
In claim 2,
The sigmoid network 60 is an input value as shown in [Equation 3] below. When entered, A method of learning a face swapping deep learning system, which is a network that generates output values of 0 to 1 based on the sigmoid function, which is a function that outputs.
[Equation 3]

In claim 2,
(c'-2) When the function of the generated attention mask 310 is M, the second synthetic conversion data 320 is calculated using the following [Equation 4]; Face swapping dip including How to learn learning systems.
[Equation 4]

(However, X is the target data 200,
delete delete