KR20240053915A - Facial expression compensation method for generating character's own facial expression and facial expression production apparatus using the same - Google Patents

Abstract

A facial expression correction method for generating a character's unique expression and an expression generating device using the same are disclosed. For this purpose, generating a plurality of first blend shapes that generate expressions for each element of the face from 3D facial expression data, and generating a plurality of first blend shapes in template data that forms a uniform facial shape without any individual features. Generating first expression data by applying, extracting an error frame in which an error occurs by comparing the 3D expression data and the first expression data for each frame, and applying a blend shape for correction to the first expression data of the error frame Provides a facial expression correction method including the step of generating second expression data by applying the second blend shape created by adding . According to the present invention, facial expressions can be corrected to express a character's unique expression within a short period of time.

Description

Facial expression compensation method for generating character's own facial expression and facial expression production apparatus using the same}

The present invention relates to a facial expression correction method for generating a character's unique expression and an expression generating device using the same. More specifically, the present invention relates to a character's unique expression that better expresses the facial expression error of a character created using blend shape. It relates to a facial expression correction method that corrects facial expressions by automatically adding blend shapes to enable facial expressions to be corrected, and an expression generating device using the same.

With the advancement of computer graphics technology, many virtual 3D characters are appearing in the production of movies and games. Because a character's face is the best way to express the characteristics of a character in a movie or game, technologies are being developed to create various and detailed facial expressions, and Blendshape is one of them. Blendshape is the most widely used method to create facial expressions of characters, and is especially used in the film industry to create realistic humanoid characters.

Blendshape refers to a set of major facial poses that are key to facial expressions, and various types of blendshapes are needed to create various facial expressions that express human emotions. Previously, people created blend shapes for each facial expression manually, which took a lot of time. To solve this problem, a technique was mainly used to create new facial expressions by linearly combining the basic shape of an expressionless face and blend shapes corresponding to various facial expressions.

Recently, attempts have been made to reduce work time by automatically creating blend shapes, but if the automatically calculated results are applied directly, fine muscles cannot be expressed properly, which reduces the completeness of facial expressions. In addition, since each character has different facial characteristics, it becomes difficult to properly express the character's unique facial expressions if the same blend shape is applied uniformly, even though the type and number of blend shapes required are diverse.

Republic of Korea Patent No. 10-217044 (2020.10.21.)

Therefore, the purpose of the present invention is to provide a facial expression correction method that can correct facial expressions by automatically adding a blend shape that can express the unique facial expressions of each character, and an expression generating device using the same.

In order to achieve the purpose of the present invention described above, the facial expression correction method for generating a unique facial expression of a character of the present invention includes (a) a plurality of first blend shapes that generate facial expressions for each element of the face from 3D facial expression data; generating a; (b) generating first facial expression data by applying the plurality of first blend shapes to template data forming a uniform facial shape without any individual features; (c) extracting an error frame in which an error occurs by comparing the 3D facial expression data and the first facial expression data for each frame; and (d) generating second facial expression data by applying a second blend shape created by adding a blend shape for correction to the first facial expression data of the error frame.

Here, step (d) includes adding at least one correction blendshape to correct the error; estimating a first weight of the blend shape for correction; Generating a second blend shape based on deep learning using the first weight; estimating a second weight of the second blend shape; and generating the second facial expression data by applying the second weight.

Meanwhile, the step of adding at least one blendshape for correction is a step of determining which facial expression to correct according to the error between the 3D facial expression data and the first facial expression data and adding a blendshape for correcting the facial expression. It is characterized by

In addition, the step of generating the second blend shape based on deep learning is characterized by using a linear regression method.

The step (c) is characterized by extracting the frame with the largest mean square error (MSE) value between the 3D facial expression data and the first facial expression data as the error frame.

And the step (b) includes generating template data that forms a uniform face shape; generating temporary facial expression data by applying the plurality of first blend shapes to the template data; Comparing the 3D facial expression data and the temporary facial expression data to determine whether an error has occurred; And when an error occurs, modifying the temporary facial expression data and comparing it with the three-dimensional facial expression data.

Meanwhile, the facial expression generation device to which the facial expression correction method of the present invention is applied generates 3D image data for the face of an actor photographed externally and then generates a plurality of first blend shapes that generate expressions for each element of the face. 3D image input unit; a first expression generator for generating first expression data for each frame by applying the plurality of first blend shapes to template data constituting a uniform facial shape without any individual features; an error frame extractor that compares the first facial expression data and the three-dimensional facial expression data for each frame to extract an error frame with the largest error; and a second facial expression generator that generates second facial expression data by applying a second blend shape generated by adding a blend shape for correction to the first facial expression data of the error frame.

Using the facial expression correction method for creating a character's unique expression of the present invention, errors can be corrected quickly by automatically adding a correction blend shape to the facial expression to which the existing blend shape has been applied, thereby reducing work time. there is.

Additionally, you can create natural changes in facial expressions in videos by automatically adding a blend shape for correction to the error frame with the largest error.

In addition, you can improve the completeness of the facial expression you want to express by selectively adding a correction blend shape that can express the character's unique characteristics.

1 is a flowchart illustrating a facial expression correction method according to an embodiment of the present invention;
Figure 2 is a diagram showing the facial shape generated at each stage according to the facial expression correction method according to an embodiment of the present invention;
Figure 3 is a diagram for explaining the process of generating first facial expression data in the facial expression correction method of Figure 1;
Figure 4 is a diagram for explaining the process of generating second facial expression data in the facial expression correction method of Figure 1;
Figure 5 is a diagram showing an expression related to the blend shape added in the facial expression correction method according to an embodiment of the present invention;
Figure 6 is a diagram for explaining the linear regression method performed in the second facial expression data generation process of Figure 4;
Figure 7 is a diagram comparing errors between data generated in the facial expression correction method according to an embodiment of the present invention, and
Figure 8 is a block diagram showing the schematic configuration of an expression generating device using a facial expression correction method according to an embodiment of the present invention.

The features and effects of the present invention described above will become more apparent through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art will be able to easily implement the technical idea of the present invention. You will be able to. Since the present invention can be subject to various changes and have various forms, specific embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to a specific disclosed form, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention.

Hereinafter, with reference to the accompanying drawings, a facial expression correction method and an expression generating device using the same for generating a character's unique expression according to a preferred embodiment of the present invention (hereinafter referred to as 'facial expression correction method and expression generating device') will be described in detail. Explain.

1 is a flowchart for explaining a facial expression correction method according to an embodiment of the present invention.

According to Figure 1, first, a plurality of first blend shapes are generated from 3D facial expression data (S100). 3D facial expression data refers to the geometry of the actor's face captured through a photography device, and the first plurality of blend shapes is data that expresses the expression of each element of the actor's face, such as the eyes, nose, and mouth.

3D facial expression data is created by overlaying a mesh filter on the actor's face recognized through a filming device, and 60 frames per second are captured. In one embodiment of the present invention, 4500 frames are acquired while the actor makes various facial expressions.

Additionally, in one embodiment of the present invention, 52 blend shapes are generated from 3D facial expression data of an actor making various facial expressions. The 52 blend shapes include 23 representing the mouth, 5 representing the eyebrows, 4 representing the chin, 3 representing the cheeks, 2 representing the nose, 1 representing the tongue, and both eyes. Each of the 7 blend shapes represented are included.

The facial geometry structure is expressed by linearly combining the blend shape and the weight of the blend shape, and this relationship can be expressed as [Equation 1] below.

는 아이덴티티(Identity;ID),

는 얼굴 기하학 구조,

는 id 계수의 벡터,

는

행렬이며, 행렬에서 열벡터는 각각의 블렌드쉐입이다. 아래 [수학식 2] 및 [수학식 3]에는 표기법을 간단히 나타내었다.In [Equation 1]

is identity (ID),

is the face geometry,

is a vector of id coefficients,

Is

It is a matrix, and the column vectors in the matrix are each blendshape. The notation is simply shown in [Equation 2] and [Equation 3] below.

After generating a plurality of first blend shapes, first facial expression data is generated by applying the plurality of first blend shapes to the template data (S200). Template data is data that constitutes a uniform facial shape without any individual features. 52 blend shapes are applied to these template data to generate first facial expression data for each of 4,500 frames. The process of generating the first facial expression data will be described in detail later with reference to FIG. 3.

When the first facial expression data is generated, the 3D facial expression data and the first facial expression data are compared for each frame (S300) to determine whether an error has occurred (S400). That is, for 4,500 frames, the 3D facial expression data and the first facial expression data are compared for each frame to determine whether an error has occurred. The occurrence of error can be determined using the mean squared error (MSE) between the 3D facial expression data and the first facial expression data. The MSE value is easily used to determine whether errors in mouth shape occur, and the larger the MSE value, the greater the error in mouth shape.

Then, the error frame with the largest error is extracted (S500). That is, the frame with the largest MSE value between the 3D facial expression data and the first facial expression data is extracted as the error frame. kth frame with largest error can be extracted using [Equation 4] below.

In [Equation 4] is the captured 3D facial expression data, is the first facial expression data with 52 blend shapes applied.

When the error frame is extracted, the second blend shape is applied to the first expression data of the error frame to generate second expression data (S600). The second blend shape is created by adding the 53rd blend shape for correction to the 52 first blend shapes, and the process of generating the second facial expression data will be described in detail later with reference to FIG. 4.

Figure 2 is a diagram showing the facial shape created at each stage according to the facial expression correction method according to an embodiment of the present invention.

first shape represents the 3D facial expression captured in step S100 and is ultimately the target facial expression to be reproduced. and is the template expression generated in step S200, The first facial expression created in step S300, corresponds to the second facial expression generated in step S600.

expressionless template expression The first look is when 52 blend shapes are applied to This is created. As shown in Figure 2 in the shape of the mouth There is a big difference with and In order to correct the error between the 53rd blend shape, the final is created.

FIG. 3 is a diagram for explaining the process of generating first facial expression data in the facial expression correction method of FIG. 1.

According to Figure 3, first, template data forming a uniform face shape is generated (S210). Template data is data that forms a standard face that does not reflect the original facial shape, and only expresses the same identity, ignoring individual unique characteristics such as facial shape or eye expression.

Temporary facial expression data is generated by applying a plurality of first blend shapes to the template data (S230). Then, the 3D facial expression data is compared with the temporary facial expression data (S250) to determine whether an error has occurred (S270). If an error occurs (S270-Y), the temporary facial expression data is corrected (S280) and step S250 is repeated. Steps S250, S270, and S280 are repeated until satisfactory results are obtained through a deep learning learning process.

If no error occurs (S270-N), the temporary facial expression data is output as the first facial expression data. Steps S230 to S290 are performed every 4500 frames to generate first facial expression data for each frame.

FIG. 4 is a diagram illustrating the process of generating second expression data in the facial expression correction method of FIG. 1, and FIG. 5 shows an expression related to the blend shape added in the facial expression correction method according to an embodiment of the present invention. Figure 6 is a diagram for explaining the linear regression method performed in the second facial expression data generation process of Figure 4.

According to Figure 4, first, a correction blend shape is added to correct the error (S620). At this time, there can be one or more blend shapes added. Depending on the error between the 3D expression data and the first expression data, which expression to correct is determined and a blend shape to correct the expression is added. That is, if the error in the mouth shape is large, a blendshape is added to correct the mouth shape, and if the error in the eye shape is large, a blendshape is added to correct the eye shape. Alternatively, a blend shape can be added to correct both the mouth and eyes. In one embodiment of the present invention, a 53rd blend shape related to the facial expression shown in FIG. 5 is added to correct the mouth shape.

The 53rd blendshape is a correction blendshape added to the kth frame with the largest error. 3D facial expression data and first facial expression data It can be obtained using the difference between the values, and is expressed as [Equation 5].

Then, the weight of the added correction blend shape is estimated (S630). Weight at kth frame is 1, and additional weights can be obtained by solving the simple linear equation shown in [Equation 6] in another frame.

After this, a second blend shape based on deep learning is generated (S650), using the linear regression method as shown in FIG. 6.

In Figure 6, is the 53rd blend shape, is temporary facial expression data generated using a linear regression method by applying the 53rd blend shape. Coefficient Wj+ Captured 3D facial expression data including is used as input data. 53rd blend shape and neutral face Input shown in [Equation 7] to accurately obtain and output The RMSE (Root Mean Square Error) value of the vertices is applied as the loss function.

The linear regression method takes less time to learn than using an autoencoder. Therefore, in one embodiment of the present invention, the blend shape obtained from the linear regression method is applied.

Then, estimate the weight of the second blend shape (S670). In order to reconstruct facial expression data, the weights must be updated, and in one embodiment of the present invention, the weights are calculated using [Equation 8] to [Equation 10] below.

Is by, Is Expressed as , the weight by pseudo-inverse is It is calculated as

Step S650 and step S670 are three-dimensional facial expression data and temporary facial expression data This is repeated until the error between the two is optimally minimized, and finally, the second facial expression data to which the finally calculated weight is applied is generated (S690).

Figure 7 is a diagram comparing errors between data generated in the facial expression correction method according to an embodiment of the present invention.

Among the 4,500 frames obtained according to an embodiment of the present invention, the 10 largest MSE·10 ⁸ and the number of blend shapes are shown in [Table 1] below. The 4206th frame shows the third largest error, but as a result of adding the 53rd blend shape, you can see that the error has been reduced by 99%. This performance is also shown in Figure 7.

It can be seen in Figure 7 that as a result of adding the 53rd blend shape, the errors of all 10 frames with the largest errors were reduced.

Figure 8 is a block diagram showing the schematic configuration of an expression generating device using a facial expression correction method according to an embodiment of the present invention.

According to FIG. 8, the facial expression generating device of the present invention includes a 3D image input unit 700, a first facial expression generating unit 720, an error frame extracting unit 740, and a second facial expression generating unit 760.

When an actor's face with various expressions captured through an external imaging device is input, the 3D image input unit 700 generates 3D image data from the input actor's face and then generates a plurality of first blend shapes. In other words, a mesh filter is overlaid on the actor's face to create a facial geometric structure, and then a plurality of first blend shapes are created that express the expression of each element of the actor's face, such as the eyes, nose, and mouth.

The first expression generator 720 generates first expression data for each frame by applying a plurality of first blend shapes to template data constituting a uniform facial shape without any individual features. To explain in more detail, the first expression generator 720 generates temporary expression data by applying a plurality of first blend shapes to the template data, and then compares the 3D expression data and the temporary expression data to generate an error. In this case, modify the temporary facial expression data. This process is performed repeatedly through deep learning until the error falls below a preset value.

The error frame extractor 740 compares the first facial expression data and the 3D facial expression data for each frame and extracts the error frame in which the largest error occurs. Here, the error frame extractor 740 determines whether an error has occurred using the mean squared error (MSE) between the 3D facial expression data and the first facial expression data.

The second facial expression generator 760 generates second facial expression data by applying a second blend shape created by adding a blend shape for correction to the first facial expression data of the error frame. To explain in more detail, the second facial expression generator 760 first estimates the weight by adding a correction blendshape to correct errors in the error frame. At this time, a deep learning-based second blend shape is created using the estimated weight, and then the final weight is estimated to generate second facial expression data to which the final weight is applied.

The facial expression generator with the above configuration can correct facial expressions by automatically adding blend shapes, allowing the character's unique facial expressions to be reproduced in a short period of time.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

700: 3D image input unit 720: first facial expression generation unit
740: Error frame extraction unit 760: Second facial expression generation unit

Claims (7)

(a) generating a plurality of first blend shapes that generate expressions for each element of the face from 3D facial expression data;
(b) generating first facial expression data by applying the plurality of first blend shapes to template data forming a uniform facial shape without any individual features;
(c) extracting an error frame in which an error occurs by comparing the 3D facial expression data and the first facial expression data for each frame; and
(d) generating second facial expression data by applying a second blend shape created by adding a blend shape for correction to the first facial expression data of the error frame; generating a unique facial expression of the character, comprising: How to correct facial expressions. According to paragraph 1,
In step (d),
Adding at least one correction blendshape to correct the error;
estimating a first weight of the blend shape for correction;
Generating a second blend shape based on deep learning using the first weight;
estimating a second weight of the second blend shape; and
A facial expression correction method for generating a character's unique expression, comprising: applying the second weight to generate the second expression data. According to paragraph 2,
The step of adding at least one blendshape for correction is,
Facial expression for generating a character's unique expression, characterized in that it is a step of determining which expression to correct according to the error between the 3D facial expression data and the first facial expression data and adding a blend shape for correcting the facial expression. Correction method. According to paragraph 2,
The step of generating the second blend shape based on deep learning is,
A facial expression correction method for creating a character's unique expression, characterized by using a linear regression method. According to paragraph 1,
In step (c),
A facial expression correction method for generating a character's unique expression, characterized in that the frame with the largest Mean Square Error (MSE) value between the three-dimensional expression data and the first expression data is extracted as the error frame. . According to paragraph 1,
In step (b),
Generating template data forming a uniform facial shape;
generating temporary facial expression data by applying the plurality of first blend shapes to the template data;
Comparing the 3D facial expression data and the temporary facial expression data to determine whether an error has occurred; and
When an error occurs, correcting the temporary expression data and comparing it with the 3D expression data. A facial expression correction method for generating a character's unique expression, comprising: A 3D image input unit that generates 3D image data for an externally photographed actor's face and then generates a plurality of first blend shapes that generate expressions for each element of the face;
a first expression generator for generating first expression data for each frame by applying the plurality of first blend shapes to template data constituting a uniform facial shape without any individual features;
an error frame extractor that compares the first facial expression data and the three-dimensional facial expression data for each frame to extract an error frame with the largest error; and
A second facial expression generator that generates second facial expression data by applying a second blend shape generated by adding a blend shape for correction to the first facial expression data of the error frame.