KR102470866B1 - Retargetimg method of 3d character facial expression and neural network learning method of thereof - Google Patents

Abstract

The embodiment relates to a method for retargeting facial expressions of a 3D character and a method for learning a neural network for this purpose. The retargeting method may include receiving a source blendshape weight corresponding to a first expression of a source facial model; generating a source facial image corresponding to a first facial expression of the source facial model by rendering the source facial model based on the source blendshape weight; generating a target face image corresponding to a first facial expression of the source face model by applying the source face image to a first neural network that generates an image corresponding to the target face model; and estimating a target blend shape weight corresponding to a first expression of the target face model by applying the target face image to a second neural network for estimating a blend shape weight corresponding to the expression of the target face model.

Description

Method for retargeting facial expression of 3D character and method for learning neural network for this

The embodiment relates to a method for retargeting facial expressions of a 3D character and a method for learning a neural network for this purpose.

As various entertainment industries such as movies, animations, games, and broadcasting develop, the importance of computer graphics that enables realistic expressions is increasing day by day. As a result, it has reached a level where it is difficult to distinguish virtual characters created using computer graphics technology from real people. Since natural facial animation is one of the most important elements to portray realistic characters, several techniques for easily producing facial animation have been introduced.

There are two main methods of producing facial animation: motion capture and key framing. In the motion capture method, data is acquired using equipment that tracks real human body and face motions, and it is refined and used as facial animation values for virtual characters. On the other hand, in the key framing method, a skilled expert creates an animation by designating a facial blendshape weight value using a program dedicated to animation production.

In general, since the face blend shape structures of different character models do not match in many cases, the animation data obtained in the above way can be applied only to one model. Therefore, when applying the acquired animation data to another model, a new model or animation must be created to have the same blend shape structure. However, in the case of motion capture, equipment is very expensive and requires specialized personnel who can refine data. In addition, when keyframe animation or character models are newly created, a lot of time and specialized personnel are required, resulting in high costs. Because of this, limitations exist.

Accordingly, research on a retargeting technology that solves the above problems and applies animation data of a source character to a target model in a semantically consistent manner has been actively conducted. However, existing studies have a problem of failing to maintain the existing blendshape structure by requiring animation data pairs of a source model and a target model in the retargeting process or by creating a new mesh.

Blendshape is a technique that creates a new facial expression by linearly combining one expressionless face mesh and a face mesh with various expressions. In order to create animations according to blendshape-based face models, experts or expensive capture equipment are generally required and time-consuming. However, since different blendshapes usually have different structures, it is difficult to use animations already created for a specific blendshape in another blendshape.

Transformation-based retargeting methodology transfers the facial expression change of the source model to the target model so that the target model makes a semantically identical facial expression to the source model. However, since the transformation transfer methodology reconstructs the model mesh, it has a limitation that it is difficult to modify after transmitting the expression because the target model cannot maintain the existing blend shape structure.

When two different faces are given, face reproduction technology that synthesizes a target face that makes the same expression as a source face can be used in various fields such as VFX, dubbing, and virtual reality.

As prior literature related to this, the following papers are disclosed.

J. Song, B. Choi, Y. Seol, and J. Noh, "Characteristic facial retargeting," Computer Animation and Virtual Worlds, vol. 22, no. 2-3, pp. 2-3. 187-194, 2011.

J. Naruniec, L. Helminger, C. Schroers, and R. Weber, "Highresolution neural face swapping for visual effects," in Computer Graphics Forum, vol. 39, no. 4. Wiley Online Library, 2020, pp. 173-184

The invention according to the embodiment maintains the blendshape structure without requiring a data pair that must be manually produced using an autoencoder to be applicable to facial reenactment technology Provides a new retargeting system want to do

Through the invention according to the embodiment, firstly, while maintaining the blendshape structure, automatic retargeting is performed without manually matching animation data pairs to match semantically, and secondly, a differentiable renderer is introduced to blend from the image of the model A new method for learning the shape inference network can be proposed, and thirdly, it is possible to enable animation retargeting by comparing the rendered images of two faces in a 2D image using face reproduction technology.

Through the application of the invention according to the embodiment, even beginners without professional knowledge of face animation can easily perform retargeting.

1 is a flowchart for explaining a retargeting method in an embodiment.
2 is an outline of a pipeline to which a retargeting method is applied in an embodiment.
3 is a flowchart for explaining a method of learning a neural network for retargeting in an embodiment.
4 is an overview of a method for training a first neural network, in an embodiment.
5 is an overview of a method for training a second neural network in an embodiment.
6 is a diagram illustrating a retargeting result in an embodiment.
7 is a diagram for explaining an effect of the presence or absence of rendering loss on a rendering result in an embodiment.
8 is a diagram showing comparative experimental results in Examples.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes can be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents or substitutes to the embodiments are included within the scope of rights.

Terms used in the examples are used only for descriptive purposes and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

In addition, in the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description will be omitted.

Also, terms such as first, second, A, B, (a), and (b) may be used in describing the components of the embodiment. These terms are only used to distinguish the component from other components, and the nature, order, or order of the corresponding component is not limited by the term. When an element is described as being “connected,” “coupled to,” or “connected” to another element, that element may be directly connected or connected to the other element, but there may be another element between the elements. It should be understood that may be "connected", "coupled" or "connected".

Components included in one embodiment and components having common functions will be described using the same names in other embodiments. Unless stated to the contrary, descriptions described in one embodiment may be applied to other embodiments, and detailed descriptions will be omitted to the extent of overlap.

1 is a flowchart for explaining a retargeting method in an embodiment.

In step 110, the device receives source blendshape weights corresponding to the first expression of the source facial model.

In step 120, the device generates a source facial image corresponding to a first expression of the source facial model by rendering the source facial model based on the source blendshape weights.

In an embodiment, the source face image generated in this way may be used to estimate a target face image and a target blend shape weight.

In an embodiment, a first neural network for generating a target face image from a source face image and a second neural network for inferring a target blendshape weight from a target face image are respectively trained to retarget expressions of the source face model to the target face model. can A semantically matched target blend shape weight may be obtained from the source blend shape weight through the first neural network and the second neural network for which learning has been completed.

In step 130, the device generates a target face image corresponding to a first facial expression of the source face model by applying the source face image to a first neural network that generates an image corresponding to the target face model.

In an embodiment, a source face image may be converted into a target face image based on a face reconstruction methodology using an auto-encoder. Accordingly, the target face image output through the first neural network may have the same expression as the source face image.

The first neural network according to the embodiment may include an autoencoder, a source encoder, and a target encoder that are trained to have the same input and output for each of the source face images and the target face images.

In step 140, the device estimates target blendshape weights corresponding to a first expression of the target face model by applying the target face image to a second neural network that estimates blendshape weights corresponding to the expressions of the target face model. .

Each blend shape model of the source face model and the target face model may include a different number of blend shape weights. Thus, through the learned second neural network, the blend shape weight of the source face image and the blend shape weight of the generated target face image may have semantically the same value.

In an embodiment, the second neural network may be trained using target face images of the target face model and target blendshape weights corresponding to facial expressions of the target face images as training data. According to an embodiment, target blend shape weights inferred from images of the target face model and target blend shape weights may be learned to be the same, and the target face image of the target face model rendered using the inferred target blend shape weights may be learned to be identical to the target face images of the target face model.

2 is an outline of a pipeline to which a retargeting method is applied in an embodiment.

를 획득할 수 있다. P는 I_t를 입력으로 받아 블렌드쉐입 가중치

= P(

)를 추론할 수 있다. 상기 과정을 통해 소스 표정 가중치 w_s와 의미상 일치하는 타깃 표정 가중치

를 획득 가능하다.The source face image I _s is rendered by receiving the source facial expression weight w _s , texture T _s , and rendering parameter p _s as input, and from I _s through an auto-encoder.

can be obtained. P takes I _t as input and blendshape weight

=P(

) can be inferred. Through the above process, the target expression weight that is semantically identical to the source expression weight w _s

can be obtained

The retargeting method proposed in the embodiment is largely divided into a face reproduction step ( 130 in FIG. 1 ) and a blendshape weight inference step ( 140 in FIG. 1 ).

First, a source face image of a source face model may be rendered using a source blend shape weight given as an input. The first neural network generating the target face image may generate a target face image having the same expression from a rendered source face image of a source face model given as an input.

In an embodiment, in the process of estimating the target blonde shape weight, a target blend shape weight that semantically matches the blend shape weight of the source facial model may be obtained from the target face image generated through the first neural network using the second neural network. have.

In an embodiment, an arbitrary facial expression of one frame may be expressed as a blendshape weight vector w. Blendshape weights for facial expressions are expressed as w _s and w _t for the source face model and the target face model. w _s and w _t do not correspond semantically to each other and may have different numbers.

Using blend shape weights, a vertex model having vertex position values, normal vectors, and texture UV coordinates can be created through V(w), which is a linear combination of blend shapes of one model. The target vertex model and the source vertex model can be obtained through V _s (w _s ) and V _t (w _t ), respectively, and for all symbols used in the embodiment, subscripts s and t denote a source and a target, respectively.

In an embodiment, frontal images of the source face model and the target face model may be used for learning of the first neural network, which may be obtained through a differentiable renderer R(*). A differentiable rendering process is expressed as Equation 1 below.

[Equation 1]

In the rendering process, vertex model V for blendshape expression data w, texture T, and rendering parameter p may be used. The texture T corresponds to the U coordinate of the model vertex V. The rendering parameter p is a 10-dimensional vector consisting of the distance, elevation, and azimuth of the camera, the 3-dimensional position of the model, the scale of the model, and the position of lighting. T and p have a single value for each model. As a result of rendering, the source model's face image I _s ∈ S and the target character's face image I _t ∈ T can be constructed. S is the area of the source face image, and T is the area of the target face image. S and T correspond to subsets of region F of the humanoid face character image.

3 is a flowchart for explaining a method of learning a neural network for retargeting in an embodiment.

The learning method according to the embodiment is divided into a step of learning a first neural network and a step of learning a second neural network.

At step 310, the device obtains a first set of training data.

In an embodiment, the first training data set may include source face images corresponding to a source face model and target face images corresponding to a target face model, and the first training data is used to learn a first neural network. It can be.

In step 320, the device learns the first neural network to output an image of the target face model using the first training data set.

In an embodiment, a method for learning the first neural network will be described in detail with reference to FIG. 4 .

4 is an overview of a method for training a first neural network, in an embodiment.

로 변환하도록 학습될 수 있다.The first neural network according to the embodiment converts the source face image I _s into a target face image based on a face reproduction methodology using an auto-encoder.

It can be learned to convert to .

In an embodiment, an autoencoder E:F -> Z may encode input images I _s and I _t into latent space Z. D _s :Z -> S is a decoder that receives latent code z encoded in space Z through E as an input and reconstructs the source face image Is, and D _t : Z -> T receives z as an input and is a target face image It is a decoder that reconstructs It.

In an embodiment, the encoder may include seven convolution layers, two fully connected layers, and one pixel shuffle layer, and the decoder may include five convolution layers and four pixel shuffle layers. Can contain layers. The overall structure of the first neural network is shown in Table 1.

[Table 1]

Table 1 is an example of the structure of the first neural network, and the convolutional filter is indicated by "(#kernel size)s(#stride)". PS2 means a pixel shuffle layer having an upscale factor of 2. The two decoders, the source decoder D _s and the target decoder D _t , have the same structure.

를 재구축하기 위한 값으로 아래의 수학식 2와 같이 정의될 수 있다.As shown in FIG. 4 , in the learning step, the first neural network may be trained using a reconstruction loss L _ae that derives the same input and output as in a general auto-encoder. The loss L _ae is given when the neural network is given face images I _s and I _t that do not correspond to each other.

It can be defined as Equation 2 below as a value for rebuilding.

[Equation 2]

In an embodiment, an autoencoder and a source decoder may be trained on a source facial image, and an autoencoder and target decoder may be trained on a target facial image.

을 출력할 수 있다.As shown in FIG. 2 , the first neural network may use an auto encoder and a target decoder to retarget an expression of a source face image to a target face model. If the source face image I _s is used as input to the first neural network composed of E and D _t trained using the source face image and the target face image, the target face image of the target face model with the same expression

can output

Again, at step 330, the device obtains a second set of training data.

In an embodiment, the second training data set may include target face images corresponding to the target face model and target blendshape weights corresponding to expressions of each of the target face images.

In step 340, the device learns the second neural network by using the second training data set to output a blend shape weight corresponding to the image of the target face model.

In an embodiment, the inference target blend shape weights inferred through the second neural network from the input target blend shape weights and the target face images may be learned to be the same, and the target face images and the inference target blend shape weights The second neural network may be trained so that the inference target face images rendered from are the same.

In an embodiment, a method for learning the second neural network will be described in detail with reference to FIG. 5 .

In step 350, the device may build a neural network for retargeting by connecting the first neural network and the second neural network.

5 is an overview of a method for training a second neural network in an embodiment.

는 이미지이므로 타깃 얼굴 모델에 적용 가능한 블렌드쉐입 가중치

를 획득하기 위해서는 타깃 얼굴 이미지로부터 블렌드쉐입 가중치를 추론하는 과정이 필요하다.In an embodiment, the target face image acquired through the first neural network

is an image, so the blendshape weight applicable to the target face model

In order to obtain , a process of inferring the blendshape weight from the target face image is required.

로부터 블렌드쉐입 가중치를 획득하기 위해 블렌드쉐입 가중치를 추론하는 신경망 P에 대해서 학습할 수 있다.Depending on the embodiment, the target face image

It is possible to learn about a neural network P that infers the blend shape weight in order to obtain the blend shape weight from

The second neural network according to the embodiment is composed of six convolutional layers and five fully connected layers. The overall structure of the neural network is shown in Table 2.

[Table 2]

According to Table 2, in the structure of the second neural network, the convolutional filter is expressed as "k(#kernel size)s(#stride)".

는 타깃 영상 이미지 I_t로부터 추론된 블렌드쉐입 가중치

를 사용하여 렌더링된 이미지로,

로 나타낼 수 있다.As shown in Figure 5,

is the blendshape weight inferred from the target video image I _t

As an image rendered using

can be expressed as

w_t를 추론하는 것이다.In an embodiment, in the learning step of the second neural network, the target face image I _t of the target face model and the target blendshape weight w _t may be used as training data. The goal of the second neural network is to obtain from the target face image I _t of the target model rendered through w _t

to infer w _t .

= P(I_t)의 차이를 통해 획득할 수 있는 블렌드쉐입 가중치 손실로 다음의 수학식 3과 같이 정의된다:As shown in FIG. 5, the loss of the second neural network is the blendshape weight w _t and the value inferred by the network

= Blendshape weight loss that can be obtained through the difference of P(I _t ), which is defined as in Equation 3 below:

[Equation 3]

L _w is a blend shape weight loss that can be obtained through the difference between the blend shape weight w _t corresponding to the input target face image I _t and the value w _t = P(I _t ) inferred through the second neural network.

와 정답 값인 w_t의 차이를 줄이는 방향으로 학습이 진행될 수 있다. 따라서, 실시예에 따른 제2 신경망이 얼굴 전면부 이미지에 영향을 끼치는 가중치 값을 중점적으로 학습하도록 미분 가능한 렌더러 R(·)을 도입하여 렌더링 손실 L_r을 정의할 수 있다.When learning is performed using only L _w , only the inferred blendshape weight is applied regardless of the actual effect on the anterior face image.

Learning may proceed in a direction of reducing the difference between w and the correct answer value w _t . Therefore, rendering loss L _r may be defined by introducing a differentiable renderer R(·) so that the second neural network according to the embodiment intensively learns weight values that affect the front face image.

를 렌더링할 수 있다.

는 정답 값인 I_t와 비교되어 로스 L_r를 구축할 수 있다. L_r을 이용하여 제2 신경망은 추론한 블렌드쉐입 가중치

로 렌더링한 타깃 얼굴 모델의 타깃 얼굴 이미지가 타깃 얼굴 이미지 I_t와 같도록 유도할 수 있다.In the learning process of the second neural network, R(·) receives the inferred w _t as an input and

can render.

can be compared with the correct value I _t to construct a loss L _r . The blendshape weight inferred by the second neural network using L _r

The target face image of the target face model rendered with can be induced to be the same as the target face image I _t .

6 is a diagram illustrating a retargeting result in an embodiment.

와 타깃 얼굴 이미지(Rendered terget)는 각각 추론된 블렌드쉐입 가중치

를 사용하여 렌더링된 타깃 얼굴 모델의 이미지를 의미한다.In an embodiment, a source face image (Rendered source) means an image of a source face model rendered using a source blend shape weight w _s ,

and the target face image (Rendered terget) are each inferred blendshape weight

It means the image of the target face model rendered using .

= D_t(E(I_s))를 획득할 수 있다. 도 6에 도시된 바와 같이 입력된 소스 얼굴 이미지 I_s는 제1 신경망을 통해 의미상 같은 표정을 가진 타깃 얼굴 이미지

로 변환되는 것을 확인할 수 있다.In an embodiment, using the encoder E and decoder D _t of the first neural network learned through I _s and I _t

= D _t (E(I _s )). As shown in FIG. 6, the input source face image I _s is a target face image having the same facial expression semantically through the first neural network.

You can check that it is converted to .

For learning according to the embodiment, 14,532 source face images and 16,050 target face images were used as training data, and the size of the images is 128 Х 128 Х 3.

는 52차원 벡터이다. 도 6의 왼쪽으로부터 네 번째 열

는 P의 입력으로 사용되어 왼쪽으로부터 다섯 번째 열과 같은 결과를 렌더링할 수 있는 블렌드쉐입 가중치

를 출력할 수 있다. 출력된

를 타깃 얼굴 모델에 적용하여 렌더링된 결과는 왼쪽으로부터 여섯 번째, 일곱 번째 열과 같다. In an embodiment, the second neural network may be taught to infer the blendshape weight w _t from I _t . Since the target face model used in the example has a total of 52 blend shapes, the inference result value

is a 52-dimensional vector. 4th column from the left in Fig. 6

is a blendshape weight that can be used as an input to P to render a result like the fifth column from the left.

can output output

The results rendered by applying to the target face model are shown in the sixth and seventh columns from the left.

7 is a diagram for explaining an effect of the presence or absence of rendering loss on a rendering result in an embodiment.

를 통해 렌더링된 타깃 얼굴 이미지, 그리고 입력 타깃 얼굴 이미지와 예측된 타깃 얼굴 이미지의 차이를 나타내고 있다.According to FIG. 7, from the left, the target face image used as an input value and the blend shape weight predicted through the second neural network

It shows the target face image rendered through , and the difference between the input target face image and the predicted target face image.

In order to prove the effect of the presence or absence of rendering loss L _r on the inference result of the neural network, blendshape weights inferred from 780 target model images that were not used for learning were constructed through the blendshape weight inference network. Quantitative evaluation can be performed by comparing the rendered image with the input image using the constructed weights, and the results can be shown in Table 3 below.

[Table 3]

Table 3 and FIG. 5 show how L _r and L _w affect the inference result when learning the blendshape weight inference network. As shown in Table 3, it can be confirmed that good results can be obtained in quantitative values when both L _w and L _r are used. In addition, FIG. 7 shows a result with a smaller difference from the input in the eyes and mouth of the face model when L _w and L _r are used together.

8 is a diagram showing comparative experimental results in Examples.

In an embodiment, a comparison with UNIT, which is widely used as an image conversion method, can be performed to compare the performance of the first neural network. UNIT shares a latent space with the first neural network used in the embodiment, but does not completely share an encoder. 128 Х 128 Х 3 size 14,532 source face images and 16,050 target face images are used for learning. UNIT was trained 250,000 times with the default settings set by the authors of the study.

8 shows the image conversion result of the first neural network and UNIT. The first column from the left shows the source face image, the second column shows the result using the first neural network proposed in the embodiment, and the third column shows the result using UNIT. It can be seen that the result using UNIT does not maintain the meaning in the part marked in red, unlike the embodiment.

In an embodiment, the retargeting method may be executed by a device (not shown). A device according to an embodiment may include a memory and a processor, and may include a program executed by the processor for retargeting. In an embodiment, the program may include the retargeting method described with reference to FIGS. 1 and 2 .

Also, an apparatus (not shown) for performing a method of learning a neural network for retargeting may be provided. An apparatus according to an embodiment may include a memory and a processor, and may include a program executed by the processor to learn a neural network. In an embodiment, the program may include a method for learning a neural network.

Through the retargeting method according to the embodiment, it is possible to propose a new type of face animation retargeting pipeline capable of retargeting even without facial expression data corresponding one-to-one between a source face model and a target face model. After rendering the source face image with input source blendshape weights by using deep learning-based face reproduction technology and converting them into target face images, blendshape weights can be inferred from images by learning a neural network for blendshape weight inference. In the learning process of the neural network for blendshape weight inference, a differentiable renderer is introduced and the inferred weight focuses on learning the value that greatly affects the front part of the actual face, thereby improving the visual result. can Through this learning process, it is possible to design a pipeline for generating blend shape weights of the target face model from blend shape weight data of the source face model.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (21)

In the retargeting method of the device for retargeting,
Receiving a source blendshape weight corresponding to a first expression of a source face model;
generating a source facial image corresponding to the first facial expression of the source facial model by rendering the source facial model based on the source blendshape weight;
generating a target face image corresponding to the first facial expression of the source face model by applying the source face image to a first neural network that generates an image corresponding to the target face model; and
Estimating a target blend shape weight corresponding to the first expression of the target face model by applying the target face image to a second neural network for estimating a blend shape weight corresponding to the expression of the target face model.
including,
An encoder and a first decoder are trained to output source face images of the source face model by inputting source face images of the source face model as training data, and inputting target face images of the target face model as training data to generate the target face images of the target face model. The encoder and the second decoder are trained to output target face images of the face model,
The first neural network includes the encoder and the second decoder,
Retargeting method.
According to claim 1,
The first neural network,
Target face images of the target face model are input as training data, and target face images of the target face model are learned to be output.
Retargeting method.
According to claim 1,
The second neural network,
Learning using target face images of the target face model and target blend shape weights corresponding to respective facial expressions of the target face images as learning data,
Retargeting method.
According to claim 3,
The second neural network,
Target blend shape weights inferred from the images of the target face model are learned to be the same as the target blend shape weights,
The target face images of the target face model rendered using the inferred target blend shape weights are learned to be the same as the target face images of the target face model.
Retargeting method.
According to claim 1,
The step of generating the source face image,
generating a vertex model by linearly combining blendshape weight vectors for facial expressions of the predetermined source face model; and
Rendering the source face model using the vertex model
including,
Retargeting method.
According to claim 1,
The source face model and the target face model include a front image of a face,
Retargeting method.
A method for learning a neural network in an apparatus for learning a neural network for retargeting,
obtaining a first training data set;
learning a first neural network to output an image of a target face model using the first training data set;
obtaining a second training data set;
learning a second neural network to output a blend shape weight corresponding to an image of the target face model using the second training data set; and
Connecting the first neural network and the second neural network
including,
The first training data set,
source facial images corresponding to the source facial model; and
Target face images corresponding to the target face model
including,
How to train a neural network.
delete According to claim 7,
Learning the first neural network,
learning an auto encoder and a source decoder such that the source face images are the same for input and output; and
Learning the autoencoder and target decoder so that the target face images are the same as input and output.
including,
How to train a neural network.
According to claim 7,
The second learning data,
Target face images corresponding to the target face model and target blendshape weights corresponding to expressions of each of the target face images
including,
How to train a neural network.
According to claim 10,
Learning the second neural network,
learning the second neural network so that the input target blend shape weights and inferred target blend shape weights inferred through the second neural network from the target face images are the same; and
Learning the second neural network so that inference target face images rendered from the target face images and the inference target blend shape weights are identical.
including,
How to train a neural network.
A computer program stored in a computer readable medium to be combined with hardware to execute the method of any one of claims 1 to 7 and 9 to 11.
In the device for retargeting,
one or more processors;
Memory; and
one or more programs stored in the memory and configured to be executed by the one or more processors;
said program,
Receiving a source blendshape weight corresponding to a first expression of a source face model;
generating a source facial image corresponding to the first facial expression of the source facial model by rendering the source facial model based on the source blendshape weight;
generating a target face image corresponding to the first facial expression of the source face model by applying the source face image to a first neural network that generates an image corresponding to the target face model; and
Estimating a target blend shape weight corresponding to the first expression of the target face model by applying the target face image to a second neural network for estimating a blend shape weight corresponding to the expression of the target face model.
and
An encoder and a first decoder are trained to output source face images of the source face model by inputting source face images of the source face model as training data, and inputting target face images of the target face model as training data to generate the target face images of the target face model. The encoder and the second decoder are trained to output target face images of the face model,
The first neural network includes the encoder and the second decoder,
Device.
According to claim 13,
The first neural network,
Target face images of the target face model are input as training data, and target face images of the target face model are learned to be output.
Device.
According to claim 13,
The second neural network,
Learning using target face images of the target face model and target blend shape weights corresponding to respective facial expressions of the target face images as learning data,
Device.
According to claim 15,
The second neural network,
Target blend shape weights inferred from the images of the target face model are learned to be the same as the target blend shape weights,
The target face images of the target face model rendered using the inferred target blend shape weights are learned to be the same as the target face images of the target face model.
Device.
An apparatus for learning a neural network for retargeting,
one or more processors;
Memory; and
one or more programs stored in the memory and configured to be executed by the one or more processors;
said program,
obtaining a first training data set;
learning a first neural network to output an image of a target face model using the first training data set;
obtaining a second training data set;
learning a second neural network to output a blend shape weight corresponding to an image of the target face model using the second training data set; and
Connecting the first neural network and the second neural network
including,
The first training data set,
source facial images corresponding to the source facial model; and
Target face images corresponding to the target face model
including,
Device.
delete According to claim 17,
Learning the first neural network,
learning an auto encoder and a source decoder such that the source face images are the same for input and output; and
Learning the autoencoder and target decoder so that the target face images are the same as input and output.
including,
Device.
According to claim 17,
The second learning data,
Target face images corresponding to the target face model and target blendshape weights corresponding to expressions of each of the target face images
including,
Device.
According to claim 20,
Learning the second neural network,
learning the second neural network so that the input target blend shape weights and inferred target blend shape weights inferred through the second neural network from the target face images are the same; and
Learning the second neural network so that inference target face images rendered from the target face images and the inference target blend shape weights are identical.
including,
Device.

Images