KR20120137826A - Retargeting method for characteristic facial and recording medium for the same - Google Patents

Abstract

PURPOSE: A face animation retargeting method and recording medium for the same are provided to obtain natural expression corresponding to the expression of source animation by similar target model by combination of an RBF based recursive training result and a kCCA based recursive training result. CONSTITUTION: A system inputs a source animation. The system sets up an expression pair range between the source animation and a target animation(S210). The system performs RBF(Radial Basis Function)-based analysis training about predetermined expression pair range(S220). The system performs kCCA(kernel Canonical Correlation Analysis)-based regression training about the predetermined expression pair range(S230). The system combines an RBF-based analysis training result and a kCCA-based analysis training result by blending parameters. The system generates the target animation expression(S240). [Reference numerals] (S210) Expression pair range setup; (S220) RBF training; (S230) KCCA based return; (S240) Final retargeting

Description

Retargeting method for characteristic facial and recording medium for the same}

The present invention relates to a method, apparatus and system for retargeting facial animation, and more particularly to a recording medium therefor. In particular, the present invention relates to a method, apparatus and system for facial animation retargeting that can be a natural face even when a face of a target character is very different from a reference character. The present invention relates to a recording medium for this purpose.

There are many ways of human communication, but one important way is facial expression. In addition, the communication method using the facial expression is similarly applied to the digital characters appearing in movies or animations. In other words, because the viewer receives a lot of information from the facial expression of the character, the character of the unnatural look greatly degrades the quality of the movie or animation. Therefore, creating an animation with a good facial expression and creating a reliable and attractive character is the most important factor when making a movie or animation in which a virtual character appears.

Facial Motion Capture & Retargeting is a technique for creating facial animation efficiently. Facial motion capture is a computerized feature that quantifies the features of real human facial expressions and fine gestures. It is a technique that enables the application of. In other words, the face motion capture technique includes important elements that make the motion look realistic such as each person's own style, mood, emotional state, weight, physical characteristics, etc., to increase the efficiency and productivity of animation work. It is a technology that can.

The face motion retargeting technique is a technique for generating animation by transplanting motion captured facial expression motion data into another virtual non-human character. When a person wants to create a facial expression by anthropomorphizing a virtual creature or animal like a character in a recent movie or an animation, since motion capture data cannot be obtained directly, face motion retargeting technique is applied to create a face animation. .

However, conventional facial motion retargeting techniques focused on reproducing realistic facial movements with high precision. That is, the aim is to reproduce the reference face motion generated by the face motion capture with the expression of a character similar to the face of a real person. This conventional face motion retargeting technique may be effective when the target character to be reproduced with the animation is similar to the face of a real person, but is not suitable for the face of a cartoon-style exaggerated character or a person who is not a person. In addition, the conventional face motion retargeting technique has a problem that the timing of the motion is too realistic to express the unique personality of a cartoon-style exaggerated character or anthropomorphized character.

Therefore, in order to apply the facial expressions of an exaggerated character or anthropomorphic character rather than a person to a movie or animation with high quality, it still depends on manual labor.

An object of the present invention is to provide a facial animation retargeting method that can automatically generate a personal facial expression of a character that is different from the actual human face.

The facial animation retargeting method for achieving the above object includes inputting a source animation as input data, setting a range of facial expression pairs between the source animation and a target animation, and a radial basis function (RBF) for the set facial expression range. A step of performing based analysis training, a step of performing Kernel Canonical Correlation Analysis (kCCA) based regression training for the set facial expression pair range, and a blending parameter in which the RBF based regression training result and the kCCA based regression training result are designated by a user And a final retargeting step that is combined by to generate an expression of the target animation corresponding to the expression of the source animation.

The step of inputting a source animation for achieving the above object is characterized in that a blendshape of a pre-captured and properly retargeted human body model is applied as the input data.

The setting of the expression pair range for achieving the above object may include setting, by an animator, a section in which a source animation and a target animation have semantically identical facial expressions as the expression pair range, and the expression pair range as a determinant. Characterized in that the step of converting.

The step of performing RBF based regression training to achieve the above object is characterized in that it comprises the step of calculating the RBF weight, and the RBF output vector using the RBF weight.

)를 이용하여 kCCA 출력 벡터가 계산되는 단계를 구비하는 것을 특징으로 한다.KCCA-based regression training step to achieve the object is a step of calculating a correlation coefficient (ρ) by using the expression matrix (S) of the source animation and the expression matrix (T) of the target animation, standard linear Generating a retargeting model to minimize errors in the least squares determination using regression, calculating a kCCA linear operator (M _kCCA ) from the generated retargeting model, and the kCCA linear operator (M _kCCA ) and the kernel Kernelized vectors obtained using functions

Is calculated using the kCCA output vector.

The facial animation retargeting method for achieving the above object further comprises the step of performing motion warping on the target animation.

In the step of performing motion warping to achieve the object, the animator sets an emotional sequence pair including both a face shape and a timing of facial expression change of the target animation by an animator for a specific emotional sequence of the source animation. Performing a heuristic search to obtain a section to which motion warping should be performed in the emotional sequence pair, and using the 1D Laplacian coordinates for the section obtained through the heuristic search. And motion warping is performed on the original retargeted motion curve suitable for the motion sections of the < RTI ID = 0.0 >

Therefore, the facial animation retargeting method of the present invention and a recording medium therefor perform the RBF based session training and the kCCA based session training once for each target model, and then the RBF based session training result and the kCCA based session training result. By combining the two, a very natural expression corresponding to the expression of the source animation can be obtained even for a very different target model morphologically. The addition of motion warping also makes the individuality of the target model more prominent. In addition, RBF-based session training and kCCA-based session training are performed only once each, and then RBF-based session training results and kCCA-based session training results are calculated by the product of simple matrix vectors, thereby generating target animations in real time. Instead of using motion capture markers directly as a source animation, it uses a mixed form of captured and retargeted motion data and an inte model, making it easy for operators to create ranges of facial expression pairs.

1 illustrates an example of a facial animation retargeting method according to the present invention.
FIG. 2 is a diagram illustrating the hybrid retargeting step of FIG. 1 in detail.
3 shows an example of a source animation and a target animation facial expression pair.
4 is a diagram illustrating a result of performing kCCA and RBF training on a source animation.
5 shows a case in which nonlinear mapping is added to a range of facial expression pairs.
6 is a diagram illustrating a result of performing kCCA and RBF training when a range of facial expression pairs is added.
7 is a conceptual diagram illustrating a method of mixing two target animations generated by kCCA and RBF training to obtain a new hybrid retargeted target animation.
8 is a view comparing the results of the hybrid retargeting of the present invention with the results of kCCA-based session and RBF-based session.
9 is a graph illustrating the error rates of FIGS. 4 and 8.
10 shows a method for measuring an error rate in the present invention.
11 is a diagram comparing a target animation generated by hybrid retargeting of the present invention with a source animation.
12 is a view comparing the target animation generated by the hybrid retargeting of the present invention with the existing NNLS technique.
FIG. 13 is a diagram illustrating the motion warping step of FIG. 1 in detail.
14 shows an example of an emotion sequence pair according to the present invention.
15 illustrates a difference in Laplacian motion warping due to interval and non-section constraints.
Figure 16 shows a target animation with one-dimensional Laplacian warping according to the present invention.
17 illustrates an example in which the hybrid retargeting technique of the present invention is applied to various characters.

In order to fully understand the present invention, operational advantages of the present invention, and objects achieved by the practice of the present invention, reference should be made to the accompanying drawings and the accompanying drawings which illustrate preferred embodiments of the present invention.

Hereinafter, the present invention will be described in detail with reference to the preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention can be implemented in various different forms, and is not limited to the embodiments described. In order to clearly describe the present invention, parts that are not related to the description are omitted, and the same reference numerals in the drawings denote the same members.

Throughout the specification, when an element is referred to as " including " an element, it does not exclude other elements unless specifically stated to the contrary. The terms "part", "unit", "module", "block", and the like described in the specification mean units for processing at least one function or operation, And a combination of software.

1 illustrates an example of a facial animation retargeting method according to the present invention.

The facial animation retargeting method according to the present invention includes a source animation input step (S100), a hybrid retargeting step (S200), and a motion warping step (S300). Implement on the face.

The source animation input step S100 is a step of receiving facial expression data of a human face animation as input data.

In general, the facial animation retargeting method uses a motion capture device to track the motion of an object and apply the motion to a virtual character. The motion capturing apparatus acquires an image of a target object by using a camera, which is an image capturing means, and uses an image that tracks a movement therefrom and applies the movement to a virtual character. Motion capture technology enables the use of computers to quantify the features of real human facial expressions and subtle gestures and apply them to virtual characters. Each of the individual's own styles, moods, emotional states, weights, physical characteristics, etc. are included in the captured motion data to increase the efficiency and productivity of the animation.

Most of the current motion capture systems are equipped with markers or sensors on the human body or face, and then analyze the marker images captured by the camera or data from the sensors to determine the position and direction of each joint or part of the human body. Use a motion capture marker to measure.

The retargeting technology is a technology for generating animation by implanting motion-captured facial expression motion data into a virtual non-human character. When retargeting a virtual creature or animal, a motion expression is directly applied. In this case, a facial expression animation is obtained by applying a retargeting technique.

However, unlike the general face animation retargeting method, the source animation input step S100 of the present invention does not directly use a motion capture marker as input data. Instead, the source animation input step (S100) of the present invention uses a blend shape of the human body model as input data in advance and is used together with appropriately retargeted motion data. This helps the animator to intuitively generate a range of expression pairs and also has the advantage of allowing any facial motion data to be used as a source in the present invention if it is configured as a blendshape. to provide.

When the source animation as input data is input through the source animation input step S100, the hybrid retargeting step S200 is performed. The hybrid retargeting step (S200) is a step of retargeting an expression of a source animation to a target animation. In the present invention, a radial basis function (RBF) and a kernel canonical correlation analysis (kCCA) based regression technique are used. We propose and use a hybrid retargeting technique that performs a combination of retargeting operations.

The hybrid retargeting technique is a technique that allows a facial expression of a target animation having a characteristic characteristic to express a very suitable expression according to the original facial expression of the input source animation. Details will be described later.

By performing the hybrid retargeting step S200, when the facial expression change of the source animation is applied to the facial expression of the target animation, a motion warping step S300 is performed to create a stylish character. The motion warping step S300 adjusts the timing at which the facial expression changes so that the facial expression change of the character may reflect the characteristics of the character.

In other words, the facial animation retargeting method according to the present invention applies a source animation to the target animation using a hybrid retargeting technique, and then adjusts the timing of the facial expression change using a motion warping technique. As a result, the facial expression of the target animation having character characteristics can be changed very naturally.

FIG. 2 is a diagram illustrating the hybrid retargeting step of FIG. 1 in detail.

As described above, the face motion retargeting technique is a technique for generating animation by implanting facial expression motion data captured by the face motion capture technique into another virtual character. The facial motion retargeting technique of the present invention can supplement the characteristics of the target character while maintaining the original motion of the motion captured facial expression motion data, which is source input data, in order to generate the facial motion of the target character. Make sure

Accordingly, the hybrid retargeting step (S200) of FIG. 2 first sets a range of facial expression pairs between semantically similar face shapes even though they are morphologically different (S210).

Here, morphologically different means a unique personality, such as a dog's face or a monster's face, in a broad sense, such as the difference between a face and a person's face. The difference between the faces can be seen as synonymous. And semantically similar face shape means that a face shape that expresses an expression such as a smiley face or a surprised face represents the same face even if the face of a monster or the face of a dog is morphologically different from the face of a person.

3 shows an example of a source animation and a target animation facial expression pair.

In FIG. 3, the upper human body model is a source animation which is motion data properly captured and pretargeted with blend shapes of the human body model, and the lower human body is a target animation corresponding to each expression of the source animation. As shown in Fig. 3, the source animation and the target animation are morphologically similar to human and monster faces, but similar face shapes representing semantically identical emotions are set as facial expression pairs.

In the present invention, 34 facial expression pairs are used as an example. In the present invention, data using a motion capture marker is not directly used.

The quality of the retargeted face animation basically depends on how the facial expression change of the source animation corresponds to the target animation. Therefore, it is important that the range of the expression pairs of the source animation and the target animation is set, and the expression pair represents a section in which the source animation and the target animation have semantically identical facial expressions. At this time, it is assumed that the source animation has a sufficient number of blend shapes, and various expressions that may be required by the animator can be described.

The range of the facial expression pair may be set by Equation 1.

Where b _i is a blend shape of the corresponding weights w _i , and Source _exp and target _exp represent semantically identical facial expressions in the source animation and the target animation, respectively. Ns denotes the number of blend shapes of the source animation, and nt denotes the dimension of the control vector cv which is an element of the target animation.

If the target animation uses non-blendshape features, such as the skeleton-driven method, the expression of the target animation is more elements represented by the control vector (cv) of the target animation. It is created with and transformed geometrically. In this way all numerical factors can be encoded into elements (cvs).

If the animator manually sets the range of the facial expression pairs, the facial expression pair ranges of the source animation and the target animation are represented by matrices as shown in Equations 2 and 3, respectively.

In Equation 2, S represents a range of the facial expression matrix of the source animation, and T in Equation 3 represents a range of the facial expression matrix of the target animation corresponding to the range S of the facial expression matrix of the source animation. And np represents the number of expression pair ranges.

When the range of facial expression pairs is set by the animator, hybrid retargeting techniques are applied in earnest. In the present invention, the hybrid retargeting technique first performs an RBF training step (S220). RBF training is represented by Equation 4 as one of powerful interpolation techniques for obtaining the best weight value between the data of the source animation and the data pair of the target animation.

In Equation 4, the variable r _i represents the RBF weight to be calculated, np represents the number of expression pair ranges, w _i represents a weight input vector, and F (w _i ) represents the calculated output. In Equation 4, the function φ _j (w _i ) is a multiple quadratic function expressed as Equation 5.

Here, the interval s _j represents the interval measured between the weight input vector w _j and the nearest weight input vector w _i , so that the feature points are transformed less if they are widely scattered, If they are located closely, they will be more deformed. Details of the feature points, along with Equation 5, can be found in "Expression cloning. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (J. Noh and U. Neumann., ACM, 2001. 383290277-288.)". As described, detailed descriptions are omitted here.

RBF training consists of source animations and control vectors (cv _i (i = 1, 2, ...) corresponding to weighted input vectors (w _i (i = 1, 2, ..., ns), where ns is a natural number). nt), where nt is a natural number) and trains np (where np is a natural number) each.

By the RBF training, a linear operator (M _RBF = [r ₁ │r ₂ │ ... r _i │ ... r _nt ]) can be obtained. Where r _i is the i-th np-dimensional weight vector.

The output vector may be calculated using the linear operator M _RBF as shown in Equation 6.

However, RBF-based regression does not perform well when nonlinearly corresponding to data samples. If an animator sets the range of facial expression pairs, if the facial expressions of two nearly identical source animations are semantically mapped to different target facial expressions, they will cause a crash in the training phase.

In the present invention, after the RBF training, kCCA based regression step (S230) is performed. CCA is a statistical analysis method used to define correlations between two sets of multivariate data. KCCA is a nonlinear statistical analysis method that uses a kernel function for CCA and is a well-known technique for analyzing nonlinear data pairs. In particular, kCCA can find the foundation pair {b _s , b _t } while maximizing the correlation between the reduced input and output data.

이고,

이다.By setting the facial expression pairs described above, each facial expression can be represented by exp = (s, t) by two sets of variables. here

ego,

to be.

The kCCA may calculate a correlation coefficient ρ by using the expression matrix S of the source animation and the expression matrix T of the target animation, as shown in Equation 7 below.

고,

계산된다. 그리고

와

는 각각 커널 함수(k)에 의해 특징 공간상에 샘플링되는 소스 및 타겟 행렬들이다. 커널 함수에 대한 상세한 설명과 kCCA 수식의 유도 과정은 Realtime data-driven deformation using kernel canonical correlation analysis(W.W. Feng, B.U. Kim, and Y. Yu., ACM Transactions on Graphics-TOG, 27(3):91??91, 2008.)에 설명되어 있으므로 여기서는 생략한다.Where h _s and h _t are coefficient vectors used to obtain the base pair {b _s , b _t }, respectively

However,

Is calculated. And

Wow

Are the source and target matrices sampled on the feature space by the kernel function k, respectively. A detailed description of kernel functions and derivation of kCCA equations is given in Realtime data-driven deformation using kernel canonical correlation analysis (WW Feng, BU Kim, and Y. Yu., ACM Transactions on Graphics-TOG, 27 (3): 91? 91, 2008.), so it is omitted here.

Generating a reduced data pair by kCCA then generates a retargeting model using standard linear regression. The retargeting model is generated to minimize an error in the least squares determination value as shown in Equations 8 and 9.

Where s _r and t _r are reduced data pairs.

The linear operators M _kCCA = M _t M _s H _s can be calculated using Equations 8 and 9 above. The output vector may be calculated from Equation 10 from the new input vector s _new .

)는 커널 함수(k)를 이용하여 획득 될 수 있다.Where the kernelized vector (

) Can be obtained using a kernel function (k).

4 is a diagram illustrating a result of performing kCCA and RBF training on a source animation. In FIG. 4, the left side is a source animation, the center is a target animation generated by kCCA training, and the right side is a target animation generated by RBF training. In FIG. 4, the target animation that performed the kCCA and RBF training using 34 facial expression pairs has a default error rate of 4.84 for the target animation that trained the kCCA and an error rate of 4.46 for the RBF training. RBF training shows better performance. However, if the nonlinear mapping is added as a range of facial expression pairs by the animator (or user) as shown in FIG. 5 (six facial expression pairs in FIG. 5), as shown in FIG. 6, the kCCA-based session has an error rate of 4.84 to 4.63. It can be reduced, resulting in better performance. RBF-based sessions, on the other hand, dramatically increase the error rate from 4.46 to 13.18. In other words, training using RBF-based sessions may show better performance, but RBF training may generate a large error depending on the number and shape of facial expression pair ranges.

This can be achieved by mixing two target animations generated by kCCA and RBF training.

In FIG. 2, when the RBF training step S220 and the kCCA based regression step S230 are completed, the final retargeting step S240 is performed by combining the results of the two steps.

7 is a conceptual diagram illustrating a method of mixing two target animations generated by kCCA and RBF training to obtain a new hybrid retargeted target animation.

The final retargeting step (S240) combines the results of the RBF training step (S220) and the kCCA based regression step (S230) as shown in Equation 11.

Where t _final is the final retargeted output vector, t _RBF is the regression result using the RBF operation according to Equation 6, and t _kCCA is the regression result using the kCCA operation according to Equation 10. And α for combining the regression result according to the RBF operation and the regression result according to the kCCA operation is a blending parameter. ) Can be adjusted within the range of [0, 1]. Where α is a real number

8 is a view comparing the results of the hybrid retargeting of the present invention with the results of kCCA-based session and RBF-based session. As can be seen in FIG. 8, even when six facial expression pairs are added to 34 facial expression pairs, the hybrid retargeting result according to the present invention shows an error rate of 4.03, which is lower than that of kCCA or RBF. That is, they are retargeted relatively accurately.

9 is a graph illustrating the error rates of FIGS. 4 and 8. 9 shows the result of the hybrid model generated by adding seven blending parameters α. As can be seen in Figure 9, the hybrid retargeting method according to the present invention can greatly reduce the error rate by using kCCA and RBF-based sessions.

10 shows a method for measuring an error rate in the present invention.

The method of measuring the error rate in Figure 10 is "J. Noh and U. Neumann. Expression

cloning. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 2001. 383290277-288. ", Employs a recursive processing technique similar to the evaluation method disclosed. That is, after retargeting the source animation of the human face to the target animation, retargeting to the human face again, and then source animation. The error rate can be measured by measuring the average vertex displacement between the human face and the human face generated by two retargeting.

11 is a diagram comparing a target animation generated by hybrid retargeting of the present invention with a source animation. In FIG. 10, it can be seen that the face of a monster to which the hybrid retargeting technique of the present invention is applied can express an expression semantically similar to the source animation, even though the face of the monster is morphologically very different from that of a human face.

12 is a view comparing the target animation generated by the hybrid retargeting of the present invention with the conventional Non-Negative Least Square (NNLS) technique. Fig. 11 shows a human face, which is a source animation on the left, applied to a dog's facial expression, and the existing NNSL technique shown in the middle does not express an accurate facial expression as shown in the case where morphologically very different. On the other hand, it can be seen that the hybrid retargeting method according to the present invention can express a similar expression regardless of morphological differences.

In addition, since the _retargeted model matrices M _RBF and M _kCCA in Equations 6 and 10 are calculated only once in each training step, only the multiplication of two matrix vectors is performed in the actual _retargeting execution step. Therefore, the hybrid retargeting technique according to the present invention can obtain a blended result model in real time. In addition, RBF training and kCCA training can reduce the training time by applying a low-density facial expression matrix, and the hybrid retargeting method according to the present invention significantly improves the overall calculation time by using a low-density facial expression matrix. The test confirms that RBF-based regression model and kCCA-based regression model by RBF training and kCCA training can be constructed within 20 seconds each.

In the above, RBF training and kCCA training are a kind of machine learning technique. When source animation is input to a target model on which RBF training and kCCA training are performed, only multiplication of the matrix vector by Equation 11 is performed to quickly retarget. The result can be obtained.

FIG. 13 is a diagram illustrating the motion warping step of FIG. 1 in detail.

When the hybrid retargeting step S200 is completed, a motion warping step S300 is performed so that the facial model of the target animation can express the character's unique personality, thereby extending the change in the facial expression of the target animation.

The facial movements of the target animation, with no motion warping and only hybrid retargeting, have a facial expression similar to that of the source, but do not represent dramatic poses and timings that represent the personality of the target face. In particular, cartoon style target faces include more character features such as squashes, stretches and exaggerations. Therefore, the motion warping step S300 is performed to generate a natural facial expression change that matches the characteristics of the target animation.

The motion warping step (S300) first performs a step (S310) of setting up emotional sequence pairs.

14 shows an example of an emotion sequence pair according to the present invention.

Emotional sequence pairs are also set manually by the animator, like facial expression pairs in the hybrid retargeting step (S200), but emotional sequence pairs include both facial shape and timing, as opposed to facial expression pair ranges. . The animator can manually create facial movements of the motion target animations for specific emotional sequences of the source animation (eg, 'surprise', 'cry', 'laugh'). When generating an emotional sequence pair for the face of a motion target animation, the new sequence pair is not limited to the source sequence. That is, the total number of frames for the movement of the emotional sequence pair may be different from the source sequence, and the flow of timing may vary in each case according to the sequence.

The only factor for detecting an emotion sequence pair is semantically identical. This provides a notable benefit when the target facial expression deviates significantly from the source facial expression. That is, even if the facial expressions of the source and the facial expressions of the target are very different, the facial expressions of the source and the facial expressions of the target may be set as emotional sequence pairs, if they can be recognized with semantically identical expressions.

If step S310 of setting emotional sequence pairs is performed, a heuristic search step S320 is performed. The heuristic search step S320 uses the sum of inner products for the current interval of interest (for each frame) to represent an absolute error between the movement of the source animation and the source interval of the emotional sequence pair. Given successive starting frames f _k of the entire source movement, the starting frames F _st for transformation are searched as shown in Equation 12.

은 리타게트된 소스 움직임의 현재 시작 프레임이고,

는 소스 쌍 구간의 시작 프레임이고, δ는 애니메이터 또는 사용자에 의해 지정되는 문턱값(threshold)이다. 본 발명에서는 문턱값(δ)을 1.0으로 사용하였다. 이 문턱값은 실험적으로 소스 쌍에서의 유사한 움직임 구간을 잘 탐색하는 것으로 확인되었다. 그러나 문턱값(δ)은 다양하게 조절될 수 있다. 그리고 동일한 구간 프레임들 내에 두 개 이상의 시작 프레임들(f_k)이 있으면, 간단하게 시작 프레임으로 그것들의 평균을 사용할 수 있다. 두 개 이상의 시작 프레임들(f_k)은 소스 쌍의 움직임이 매우 자연스럽게 변화될 때, 구간의 시작점에서만 야기되므로, 두 개 이상의 시작 프레임들(f_k)의 평균을 시작 프레임으로 사용하여도 무방하다.here

Is the current starting frame of the retargeted source movement,

Is the start frame of the source pair interval, and δ is the threshold specified by the animator or user. In the present invention, the threshold value δ is used as 1.0. This threshold has been experimentally confirmed to search for similar movement intervals in the source pair. However, the threshold δ can be adjusted in various ways. And if there are two or more start frames f _k in the same interval frames, then we can simply use their average as the start frame. Since two or more start frames f _k are caused only at the start of the interval when the source pair's movement changes very naturally, the average of two or more start frames f _k may be used as the start frame. .

When the most suitable section for the transformation is obtained through the heuristic search step (S320), motion warping is performed on the original retargeted motion curve suitable for the target motion sections using 1D Laplacian coordinates. The final movement is matched appropriately, still resembling the original retargeted motion even when the emotional target interval is an issue.

In the present invention, in order to preserve the original retargeted motion, the key value f (t) is transformed in all n frames for all target control dimensions nt on the one-dimensional Laplacian coordinates. F _o is an n X nt matrix that stores all kit values for each control dimension. The Laplacian coordinates L are then calculated as in Equation 13.

Here, Δ is a one-dimensional Laplacian matrix of (n-1) X n, and the elements of this matrix are constructed as shown in Equation (14).

The present invention uses Laplacian coordinates to encode all the details of the key values that change from the original retargeted movement. In addition this allows the remaining key values to be interpolated properly, allowing to include a subset of the key values in the target interval and the original retargeted movement. In the present invention, the key values F are restored from the Laplacian coordinates, and the key values F are constrained to satisfy the linear least-squares expression of Equation (15).

In this system, the m X nt matrix T includes m target interval key values in which frames are constrained according to the rows of the m X n matrix C _T having 1 in the corresponding column. Similarly, the l X nt matrix P contains the original non-interval key values of the l constrained frames, according to the l X n constraint matrix C _P. In the present invention, for the target interval constraint matrices C _T and T, all key values are constrained according to the interval frames for the emotional sequence. And in order to prevent the original retargeted motion from exceeding the target interval sequence, in the present invention, as shown in FIG. 3, an additional l constraint (l is a natural number) is added.

15 illustrates a difference in Laplacian motion warping due to interval and non-section constraints. In FIG. 15, the lower figure shows the original retargeted motion according to the target interval and target non-interval constraints that are warped to match the target interval while the original retargeted motion is preserved.

The first and third items ensure that the details of the original retargeted movement are preserved. The second item ensures that the emotional time intervals are met. In order to allow the animator or the user to modify the final result, the present invention uses soft constraints rather than hard constraints, including values other than one.

Figure 16 shows a target animation with one-dimensional Laplacian warping according to the present invention.

As shown in FIG. 16, in the target animation on which motion warping is performed, the timing of the source animation and the facial expression change are different from each other, and the character of the character may be more prominent.

17 illustrates an example in which the hybrid retargeting technique of the present invention is applied to various characters. As shown in FIG. 17, when the hybrid retargeting technique of the present invention is applied, the human facial expression of the source animation can be naturally applied to various characters having very different morphologically.

The device according to the invention can be embodied as computer readable code on a computer readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and a carrier wave (for example, transmission via the Internet). The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art.

Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

Claims (22)

Inputting a source animation as input data;
Setting a facial expression pair range between the source animation and the target animation;
Performing RBF-based analysis training on the set facial expression range;
Performing a Kernel Canonical Correlation Analysis (kCCA) based regression training on the set facial expression range; And
And a final retargeting step of combining the RBF based regression training result and the kCCA based regression training result by a blending parameter designated by a user to generate a facial expression of the target animation corresponding to the facial expression of the source animation. Way.
The method of claim 1, wherein the input of the source animation is performed.
A blendshape of a pre-captured and properly retargeted human body model is applied as the input data.
The method of claim 2, wherein the setting of the facial expression pair range is
Setting, by an animator, a section in which a source animation and a target animation have semantically identical facial expressions to the expression pair range; And
And converting the range of facial expression pairs into a determinant.
The method of claim 3, wherein the setting of the facial expression pair range
Equation in each of the source animation and the target animation

Where b _i is the blend shape of the corresponding weights (w _i ), Source _exp and target _exp represent the semantically identical facial expressions in the source and target animations respectively, ns is the number of blend shapes in the source animation , nt is the dimension of the control vector (cv) that is the element of the target animation.
Facial animation retargeting method, characterized in that set by.
The method of claim 4, wherein the step of converting into a determinant
Each of the source animation and the target animation is formula

And

Where S is the range of the expression matrix of the source animation, T is the range of the expression matrix of the target animation corresponding to S, and np is the number of ranges of the expression pairs.
Retargeting facial animation, characterized in that for converting. The method of claim 5, wherein the RBF-based regression training is performed
Calculating an RBF weight; And
And calculating an RBF output vector by using the RBF weights.
The method of claim 6, wherein the step of calculating the RBF weight is
The RBF weight is equation

(Where ri represents the RBF weight to be calculated, np represents the number of facial expression pair ranges, wi represents the weight input vector, and F (wi) represents the calculated output.)
Lt; / RTI >
The function φ _j (w _i ) is

(Where sj represents the interval measured between the weight input vector wj and the nearest weight input vector wi).
Facial animation retargeting method, characterized in that is calculated by.
8. The method of claim 7, wherein calculating the RBF output vector
The target animation corresponding to the weight input vector wi (i = 1, 2, ..., ns) and the target animation corresponding to the control vector cvi (i = 1, 2, ..., nt) Each of these np (np is a natural number) times, RBF linear operator (M _RBF )
(M _RBF = [r ₁ │r ₂ │ ... │r _i │ ... │r _nt ])
Where r _i is the i-th np-dimensional weight vector
Lt; / RTI >
The RBF output vector t _RBF is _{expressed as}

Facial animation retargeting method, characterized in that is calculated by.
The method of claim 8, wherein the kCCA-based regression training step is
Calculating a correlation coefficient p using the facial expression matrix S of the source animation and the facial expression matrix T of the target animation;
Generating a retargeting model such that an error is minimized in the least squares determination using standard linear regression;
Calculating a kCCA linear operator (M _kCCA ) from the generated retargeting model; And
Kernelized vector obtained using the kCCA linear operator (M _kCCA ) and a kernel function (

And calculating the kCCA output vector using the < RTI ID = 0.0 >
10. The method of claim 9, wherein the kCCA based regression training step
The correlation coefficient is

Where h _s and h _t are coefficient vectors, respectively

ego,

Is calculated as

Wow

Denote the source and target matrices sampled on the feature space by the kernel function k, respectively.)
Lt; / RTI >
The retargeting model is equation

And

Where s _r and t _r are reduced data pairs
Generated by
The kCCA linear operator (M _kCCA ) is a mathematical formula
Calculated by M _kCCA = M _t M _s H _s ,
The kCCA output vector t _kCCA is

Facial animation retargeting method, characterized in that is calculated by.
The method of claim 10, wherein the final retargeting step
The final retargeting output vector tfinal is the RBF output vector tRBF, which is the RBF-based regression training result, and the kCCA output vector tkCCA, which is the kCCA-based regression training result, in the blending parameter α set by the animator. By equation

(Where blending parameter α is a real number within [0, 1] range)
Facial animation retargeting method characterized in that it is calculated as.
The method of claim 11, wherein the facial animation retargeting method is
And performing motion warping on the target animation.
The method of claim 12, wherein the motion warping is performed
Setting, by the animator, an emotional sequence pair that includes both the facial shape and the timing of facial expression change of the target animation by an animator for a specific emotional sequence of the source animation;
Performing a heuristic search to obtain a section in which the motion warping should be performed in the emotional sequence pair; And
Motion warping is performed on the original retargeted motion curve suitable for the motion sections of the target animation using 1D Laplacian coordinates for the section obtained through the heuristic search. How to retarget facial animations.
The method of claim 13, wherein the heuristic search is performed
Start frames F _st to which the motion warping should be performed are

(here

Is the current starting frame of the retargeted source movement,

Is the start frame of the source pair interval, and δ is the threshold specified by the animator or user)
Facial animation retargeting method characterized in that the navigation by.
The method of claim 14, wherein the motion warping is performed using the one-dimensional Laplacian coordinates.
Converting a key value f (t) in all n frames for all target control dimensions nt on the one-dimensional Laplacian coordinates;
Laplacian coordinates (L) are equations

(Where Fo is an n X nt matrix that stores all key values for every control dimension, and Δ is a one-dimensional Laplacian matrix with (n-1) X n)
The element of Δ is

And a step calculated by the method for facial animation retargeting.
The method of claim 15, wherein the motion warping is performed using the one-dimensional Laplacian coordinates.
Restoring the key values (F) from the one-dimensional Laplacian coordinates; And
Equation

And constraining the key values (F) to satisfy the linear least squares of.
17. The computer program product of claim 1 to 16, wherein program instructions for executing the facial animation retargeting method are recorded.
Setting, by the animator, an emotional sequence pair that includes both the facial shape and the timing of facial expression change of the target animation by an animator for a specific emotional sequence of the source animation;
Performing a heuristic search to obtain a section in which the motion warping should be performed in the emotional sequence pair; And
And performing motion warping on the original retargeted motion curve suitable for the motion sections of the target animation using 1D Laplacian coordinates for the section obtained through the heuristic search. How to warp face animation motion.
19. The method of claim 18, wherein the heuristic search is performed
Start frames F _st to which the motion warping should be performed are

(here

Is the current starting frame of the retargeted source movement,

Is the start frame of the source pair interval, and δ is the threshold specified by the animator or user)
Facial animation motion warping method characterized in that the navigation by.
The method of claim 19, wherein the motion warping is performed using the one-dimensional Laplacian coordinates.
Converting a key value f (t) in all n frames for all target control dimensions nt on the one-dimensional Laplacian coordinates;
Laplacian coordinates (L) are equations

(Where Fo is an n X nt matrix that stores all key values for every control dimension, and Δ is a one-dimensional Laplacian matrix with (n-1) X n)
The element of Δ is

And a step computed by the facial animation motion warping method.
The method of claim 20, wherein the motion warping is performed using the one-dimensional Laplacian coordinates.
Restoring the key values (F) from the one-dimensional Laplacian coordinates; And
Equation

And constraining the key values (F) to satisfy the linear least squares of.
22. The computer program product of claim 18, wherein program instructions for executing the facial animation motion warping method are recorded.

Images