KR100813034B1 - Method for formulating character - Google Patents

Abstract

An animation character forming method is provided to provide more natural and smooth lipsync animation while reducing manpower consumed in producing face animation. Animation character data is received(S100). A figure model of the animation character is divided into a portion which is mainly used for pronunciation and a portion which is mainly used for expressing a look, and the portion mainly used for pronunciation is animated to form a lipsync animation of the animation character(S110). The portion mainly used for expressing a look is animated to form a look animation of the animation character in which the lipsync animation has been formed(S120,S130). The character data includes voice information and look information of the animation character.

Description

Method for formulating character

1 and 2 are a flow chart of an embodiment of a character forming method according to the present invention,

3 is an overall flowchart of a TTVS system;

4 is a diagram illustrating 11 phoneme models of phonemes of Korean vowels and consonants.

5 is a diagram showing 13 phoneme models of phonemes of English vowels and consonants,

6 is a diagram illustrating an embodiment of a Catmull-Rom spline interpolation method,

7 is a diagram illustrating a weight calculation function for correcting a model of bism,

8 is a view showing a mixture of lip sync animation and emotional expressions,

9 is a diagram illustrating an interface for mixing between an expression model representing emotion and an emotion expression,

10 is a diagram showing quantification of facial expression input showing emotion,

11 is a diagram showing primary, secondary and final pronunciation importance and facial expression importance,

12 illustrates a combination of pronunciation and facial expressions when pronunciation importance is high.

The present invention relates to a character formation method, and more particularly to a lip sync animation and facial expression animation of the character.

For realistic face animation generation, various approaches have been proposed depending on whether the target face is a 2D image or a 3D mesh model.

Regarding the lip-sync animation for the two-dimensional face image, the existing video image is analyzed, cut into short video sequences for each of three consecutive phoneme combinations, and fit into a new voice track. It has been suggested to reconnect.

However, the method described above is sample-based and can produce very close results, but a large library is needed because a large amount of data must be stored.

In addition, in the method of lip synchronization animation using two-dimensional face images as an example, a subjective subject selects a face image corresponding to an example model for each phoneme, and an intermediate mouth shape is used for image morphing for animation. A method of creating a lip synchronization animation by using the same technique has also been proposed.

However, the above-described method makes an animation with only the mouth shape image of as many as vismes, so it is difficult to express the co-articulation effect for the same pronunciation according to the front and back pronunciation. Has been.

In the lip synchronization animation targeting the 3D mesh model, a lip sync synchronized with the voice is generated by generating a sequence of phonemes by receiving the spoken contents as script text, and interpreting the information. The best way is to create an animation.

However, although the above-described method can generate a lip sync animation synchronized with the voice, a method of naturally adding emotion to the lip sync animation in order to create a more realistic facial expression has not been proposed. In particular, most of the existing lip-sync animation generation methods do not consider a co-articulation effect when the lip-sync animation is produced, even if the shape of the mouth is the same for the same pronunciation. As we add facial expressions, we do not propose an effective way to express those facial expressions in the face model without significantly breaking the appearance around the mouth for the current pronunciation.

Even when creating a lip-synced animation of a virtual character, to look more natural, we need to combine a lip sync animation generated from the TTVS system with a facial expression animation representing emotions.

The present invention has been proposed to solve the above problems, and provides a method of generating a lip-sync animation of Korean and English and naturally reflecting facial expressions representing emotions in the process of producing 3D facial animation. To generate facial animations, the phoneme information can be generated using existing text-to-speech (TTS) systems to automate the creation of lip-sync animations from the given voice and phoneme information to minimize the user's manual operation and to produce realistic results in real time. It provides a user interface for facial expression parameters that are input from a voice file and change continuously over time. In particular, an importance-based approach is proposed to combine lip-sync animation with facial expressions that express emotions.

The present invention is to solve the above problems, an object of the present invention to generate a lip-sync animation showing the emotions of the character in real time, to provide a character formation method for expressing the emotions of the character that continuously changes over time It is.

Another object of the present invention is to form a more natural and smooth lip sync animation.

Still another object of the present invention is to provide a lip sync animation of a character representing a realistic emotional expression.

Another object of the present invention is to minimize the user's manual work consumed in producing the facial animation, and to maximize the accuracy of the voice, the naturalness and real-time of the animation.

In order to achieve the above object, the present invention comprises the steps of receiving character data; Forming a lip sync animation of the character according to the material; And forming a facial expression animation on the character on which the lip sync animation is formed.

According to another embodiment of the present invention, an input unit for inputting character data; A lip sync animation forming unit for forming a lip sync animation of the character; And a facial expression animation forming unit for forming a facial expression animation in the character on which the lip sync animation is formed.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention that can specifically realize the above object will be described.

The same components as in the prior art are given the same names and the same reference numerals for convenience of description, and detailed description thereof will be omitted.

According to the present invention, when a user or the like inputs character information, an animation including an expression is formed according to the input information.

1 and 2 are flowcharts of one embodiment of a character forming method according to the present invention. An embodiment of the character forming method according to the present invention with reference to Figures 1 and 2 as follows.

As shown in FIG. 1, the voice and facial expression data of the character are input (S100), and then the lip sync animation S110 and the facial expression animation S120 of the character are sequentially formed. In addition, it is preferable to synchronize the lip sync animation and the facial expression animation described above (S130).

In FIG. 2, a lip sync animation is generated by analyzing a voice or an external voice file synthesized by a TTS system or the like with respect to the character information input in the form of script text, and then animating the expression of emotion to generate the lip sync animation described above. To combine. At this time, the face model of the character is divided into a part mainly used for pronunciation and a part mainly used for expressing an expression, and the lip sync animation animates a part mainly used for pronunciation, and the formation of a facial expression animation mainly represents an expression. It is characterized by animating the parts used.

Hereinafter, the above-described process will be described in detail.

The goal of lip-sync animation is to create a natural and realistic mouth shape for a given voice track. The voices given as inputs are voices recorded by real people or synthesized by the TTS system. In addition, to generate a lip sync animation, information about a phoneme sequence of information to be spoken and information about a phoneme duration of each phoneme is required. In the case of using the actual human upbringing, phonetic alignment systems may be required to analyze recorded voice signals and generate phoneme information. In addition, the TTS system has a disadvantage in that it feels awkward compared to the actual human development. However, since the phoneme information can be directly obtained from the TTS system, a more accurate lip synchronization animation can be produced.

The present invention proposes a method for processing phoneme information necessary for generating a lip sync animation under the assumption that a phoneme sequence and length information of an input voice are given. A system for generating a lip sync animation by receiving script text as input from a user is called a text-to-visual speech system. As shown in FIG. 3, the TTVS system performs natural word processing on a given script text to generate various phoneme information. Using the information, the voice processor synthesizes a voice and the image processor synthesizes an image. It is desirable to synchronize the generated voice and video. The TTS system not only generates synthesized speech but also provides phoneme sequence information and length information of each phoneme over time. In addition, it is possible to synchronize the synthesized voice and video using phoneme information obtained from the TTS system.

Phonemes are the smallest unit of distinction in pronunciation. When a person speaks, he pronounces the phonemes listed in sequence. In order to perform this process on a computer, the lip-sync animation lists the corresponding mouth shapes in phoneme order to be pronounced and creates a continuous movement by smoothly connecting them. The present invention proposes a method for generating a natural and realistic lip sync animation synchronized with a given voice by classifying the phonemes of the phonemes and producing a three-dimensional face example model corresponding thereto.

First, we propose how to classify and construct a phoneme example model for Korean and English. Even if you want to lip-sync a character in a language of another country, you can classify and compose according to Korean and English.

In order to produce a lip-sync animation, the present invention uses a key-framing animation technique that looks at the mouth shape of each phoneme as a key-frame for a given phoneme sequence and smoothly fills the space between them. The reason for this approach is based on the assumption that human speech can be made up of all finite models of vis. Here 'bism' generally refers to the shape of the mouth that is distinguished from others by the eye when a person makes an pronunciation. Therefore, the bismuth model refers to the face model of a virtual character having a three-dimensional mesh structure having a mouth shape that is distinct from other things when it is pronounced.

In other words, the present invention assumes that a human speech can be generated by using a finite number of bismuths, and one approach that can be applied is to define a one-to-one correspondence between the phonemes and the bisms for each phoneme. It is to make a lip-synced animation by creating a bismuth model. In this way, when the TTS system generates phoneme sequence information and phoneme length information for the text given as input, the lip sync animation is generated by placing bismuth models at corresponding points and smoothly connecting them.

First, the shape of the mouth of the short vowels play an important role in pronunciation, so they have different mouth shapes. However, in general, several phonemes can be mapped to a single bism. For example, Korean 'ㅁ' and 'ㅂ' or English 'o' or 'w' have different sounds in similar mouth shapes. Like this, in the case of consonants, there is a characteristic of making different pronunciations with a similar mouth shape, and a double vowel is a combination of a half vowel or another short vowel that cannot be used alone. Therefore, consonants are classified into several groups having the same mouth shape to correspond to one bismuth, and a double vowel is divided into two short vowels to generate an animation.

Table 1 classifies phonemes with similar mouth shapes into corresponding vowels and consonants, and Table 2 classifies English consonants and vowels into corresponding bisms.

Phoneme classification Biblical classification collection monophthong ㅏ, ㅐ, ㅓ, ㅔ, ㅗ, TT, ㅡ, ㅣ, ㅚ, ㅟ ㅏ, ㅐ, ㅓ, ㅔ, ㅗ, TT, ㅡ, ㅣ Double vowels ㅑ, ㅒ, ㅕ, ㅖ, ㅛ, ㅠ, ㅢ, ㅙ, ㅞ, ㅘ, ㅝ Consonant Research ㄱ, ㄲ, ㅋ, ㅇ A Voice ㅎ Alveolus C, k, k, k, k, b, d C Oral opening ,, ㅉ, ㅊ A positive sound ㅂ, ㅃ, ,, ㅁ M

FIG. 4 is a diagram illustrating Korean vowels composed of 40 phonemes in 11 vowel and consonant bismuth models, and FIG. 5 illustrates English constituents composed of 41 phonemes in 13 vowel and consonant bismuth models. It is a figure which shows that.

Next, a method of generating a lip sync animation using face example models corresponding to each phoneme example model will be described. That is, how to generate the lip synchronization animation using the phoneme example model described above will be described.

Here, when the TTS system generates the phoneme sequence information and the phoneme length information for the text given as an input, the BTS is arranged according to the phoneme sequence and the time length and the keyframe animation is connected between them.

In the present invention, Catmull-Rom spline interpolation is used to generate a smooth lip sync animation. Catmull-Rom splines are a series of curves defined as cubic continuity between each vertex and a vertex, given a sequence of vertices on a plane or space, with the differential coefficient at each vertex It is equal to the slope of a straight line between two neighboring vertices.

6 shows an example of the Catmull-Rom spline interpolation method. The set of vertices V ^p for the bism model p is defined as

V ^p = {V ^p ₁ , V ^p ₂ , ..., V ^p n}

V ^p _i = {x ^p _i , y ^p _i , z ^p _i }

The phoneme sequence information P and the phoneme length information L obtained from the TTS system are as follows.

P = {p ₁ , p ₂ , ..., p _m }

L = {ℓ ₁ , ℓ ₂ , ..., ℓ ₃ }

Then, the time at which the bismuth model should be located is calculated from the phoneme length information L. Assuming that each bismuth model is located in the middle of the time when the phonemes are pronounced, the position t _j on the time axis of the bismuth model p _j is calculated as follows.

The pronunciation time information T of the phoneme thus obtained can be displayed as follows.

T = {t ₁ , t ₂ , ... t ₃ }

Using the Catmull-Rom spline interpolation method, the x-coordinate of the i th vertex of the face model at any time t where t _j ≤ t <t _{j +1} can be expressed as follows.

x (t) = at ³ + bt ² + ct + d

Here, a, b, c and d are obtained from the following four conditions.

는 비즘 모델 p _j 의 i번째 정점의 x 좌표 x^pji와 같다.Here, x ^p _i is the x-coordinate of the i th vertex of the bismuth model p _j included in the phoneme sequence information P. That is, time t _j Because the model is bijeum p _j, the virtual character's face, x-coordinate of the vertex of the i-th face model

Is equal to the x coordinate x ^pj i of the i th vertex of the bism model p _j .

는 비즘 모델 p _j+1 의 i번째 정점의 x 좌표 x^{pj +1} _i와 같다. 또한, Catmull-Rom 스플라인 보간법을 사용하기 때문에 시간 tj에서 xi 의 시간에 대한 순간 변화율

는

x^{pj -1} _i에서 x^pj+1 _i으로의 시간에 대한 평균 변화율과 같다.Similarly, x coordinate of face model i th vertex at time t _{j +1}

Is equal to the x coordinate x ^{pj +1} _i of the i th vertex of the bismuth model p _{j + 1} . In addition, the rate of instant change with respect to the time of xi at time tj , because Catmull-Rom spline interpolation is used.

Is

It is equal to the average rate of change over time from x ^{pj -1} _i to x ^{pj + 1} _i .

은 x^pj _i에서 x^pj+2 _i로의 시간에 대한 평균 변화율과 같다. 상술한 같은 네 개의 조건으로부터 네 개의 미지수 a, b, c, d의 값을 결정한다. y 좌표와 z 좌표에 대해서도 같은 방법을 적용한다.Similarly, at time t _j x _{i +1} Rate of change of time

Is equal to the average rate of change over time from x ^pj _i to x ^{pj + 2} _i . The four unknowns a, b, c, d are determined from the same four conditions described above. The same applies to the y and z coordinates.

And if you make an animation using the non-machining model of the artist made by the artist, the movement of the mouth shape is not smooth. Speaking shows frequent movements at very short time intervals, and it is not smooth because it creates animations that go through all the Bism models in such a short time. In addition, the actual person pronounces the same phoneme in a slightly different mouth shape depending on the type of phoneme and the pronunciation time. This should also be taken into account when creating a lip sync animation of a virtual character.

The present invention proposes a method of modifying the mouth shape of the phoneme currently being pronounced in consideration of the mouth shape of the front and back phonemes according to the length of each phoneme.

의 위치는 아래와 같이 수정된다. I-th vertex of the j-th bijeum model pj included in the phoneme sequence information obtained from the audio file obtained P

The position of is modified as follows.

의 위치는 비즘 모델 pj-1의 i번째 정점으로부터 자기 자신까지 잇는 선분 위의 점이다. 이것은 발음되는 시간이 짧은 음소일수록 weight가 작아지게 하여 대응되는 비즘 모델의 위치까지 도달하지 못하고 다음 발음에 해당하는 비즘 모델의 위치를 향하여 애니메이션이 되도록 하기 위함이다. 여기서 weight는 발음 시간길이에 따라 선형적으로 변한다고 가정할 수도 있지만 부드럽게 보간하기 위하여, 다음과 같은 삼차식으로 정의된 함수에 따라 계산한다.That is, the weight value is modified as it changes between 0 and 1.

Is the point on the line from the i th vertex of the bismuth model pj-1 to itself. This is because the shorter the phoneme is pronounced, the smaller the weight is, so that it does not reach the position of the corresponding bismuth model and is animated toward the position of the bismuth model corresponding to the next pronunciation. In this case, the weight may be assumed to change linearly according to the pronunciation time length, but in order to smoothly interpolate, the weight is calculated according to a function defined by the following three equations.

When a person moves around the mouth and around it, he moves with strength. At this time, if you observe the moving speed, the moving speed increases initially and then decreases again when switching from the mouth shape of the currently pronounced phoneme to the mouth shape for the next pronunciation. Thus, the area around the mouth changes slowly at first, then increases in speed and then changes slowly. As a result, the phoneme to be pronounced in a very short time does not reach a relatively corresponding bismuth model, and reaches a corresponding bismuth model when there is enough pronunciation time. To express this phenomenon, we approximate the weight to change according to the function given above. The change in weight according to the phoneme time length is shown in FIG. 7. In this way, by modifying the bismuth model according to the phoneme time length and producing a lip synchronization animation by Catmull-Rom interpolation, a natural and smooth result is obtained.

The following section describes a mix of lip sync animation and emotion animation.

When a person talks, he expresses his emotions with an expression on his face and speaks simultaneously. Even when creating a lip-synced animation of a virtual character, to look more natural, we need to combine a lip sync animation generated from the TTVS system with a facial expression animation representing emotions. For example, as shown in FIG. 8, the face of an 'a' in Korean or an 'a' in English and an angry expression are combined to show an 'a' or 'a' in English in an angry face. You should be able to create We propose a method to combine facial expressions into a lip synchronization animation by using the differences in importance of pronunciation and facial expressions for each part of the face.

First, we define a facial expression example model to be mixed with lip sync animation, and propose an emotion input interface that considers the user's convenience for creating various facial expressions. Next, we propose an importance-based approach for blending emotions into lip-sync animations.

This article describes how to create a facial expression example model that shows emotions, and an emotion input interface that can be easily applied to a lip-sync animation. First, facial expression models for joy, sadness, surprise, fear, anger, etc., which are representative emotions to be expressed in the lip-sync animation, are prepared in advance, and various expressions are generated through blending between them. Here, joy, sadness, etc. are merely exemplary classifications of emotions. That is, the expression animation is characterized in that the character's face is divided into predetermined areas, and each divided area is independently animated according to the input character data. In this case, since the lips of the character's face mainly express emotions, the facial expression animation may be formed using only the data related to the lips among the input character data.

9 shows a facial expression example model for these representative emotions. At one point, the expression depends on the degree of emotional state, such as joy, sadness, surprise, fear, or anger. By expressing the state of emotion, a face corresponding to a different degree of emotion may be displayed from a face in which the present expression is not expressed. For example, to create a face that is as happy as 0.5 and sad as 0.5 when the emotional value of each facial expression model is 1, the half size and sadness of the difference in displacement between each vertex between the facial expression model that expresses joy and the expressionless expression. Such an expression is generated by adding half the magnitude of the displacement difference between each vertex between the facial expression model representing the facial expression model and the facial expressionless expression model to the expressionless facial model. In addition, by adjusting the displacement difference of each vertex between the generated facial expression model and the model representing the expressionless expression, it is possible to control the facial expression according to the emotion change.

For arbitrary facial expressions e1 and e2 , the position of the i th vertex of the new facial expression model whose ratio of emotion for both expressions is α: (1 α) and the intensity of emotion from the expressionless face is β is calculated as follows.

Where V ^N _i , V ^e1 _i and V ^e2 _i are the positions of the i th vertex of the facial expression model corresponding to the expressionless expression, the facial expression e1 , and the facial expression e2 , respectively.

Next, a user interface for easily mixing expressions will be described. The input information to be obtained from the user is e1 and e2 facial expression and emotion degree α to be mixed at each time, and the intensity β from emotionless face. Since it is difficult for a user to provide such information every frame, information is obtained for only a few key frames and filled by interpolation. To generate smooth animations, we use cubic spline interpolation to satisfy C2 continuity.

9 shows an interface for generating facial expression animation. The circle with a radius of 1 is divided into five, centered on the white dot in the center, which means no expression, and the five expressions of joy, sadness, surprise, fear, and anger are placed. That is, for n expressions to be added, the circle is divided into n parts so that all expressions are placed on the circumference. At this time, one point inside the circle means a facial expression to be newly generated, and all information necessary for generating a new facial expression model is obtained from this point.

As shown in FIG. 10, the expression models e1 and e2 are determined according to which of the five sectors are located in the sector, and the ratio of the center angles of the two sectors divided by the straight line connecting the center and the given point is α: (1 -α ) . Also, the distance between the given point and the center of the circle determines the β value. When the user takes a facial expression model corresponding to several key frames through this interface, a cubic spline interpolation method creates a curve animation using C3 continuity by using a curve and the positions of points on the curve.

Next, we describe the importance-based approach needed to blend emotions into lip-sync animations.

When a person is not speaking, he expresses his feelings through his entire forehead, eyes, eyebrows, cheeks, and mouth. However, when speaking, the lower part of the face, which corresponds to the mouth and its surroundings, is mainly used for pronunciation, and the upper part of the face, such as the forehead, eyes, and eyebrows, is used to express the expression of emotion. Therefore, in order to create a facial expression that is pronounced by making a facial expression using a facial expression that is pronounced with an expressionless face and a facial expression that expresses an expression for an emotion, the importance of pronunciation and expression for each part of the face It should be synthesized in different ways. This section proposes how to analyze pronunciation models and facial expression models to define the importance of pronunciation and facial expression for each vertex of face model and to create a new facial image combining pronunciation and facial expression based on this importance. do.

For example, a part with a lot of movement in order to pronounce the voice compared to other parts of the face, such as the lower lip or the jaw, is an important part of pronunciation. In addition, the upper lip and the lower lip compared to the lower jaw, but the movement for pronunciation is a part of high importance for pronunciation. Therefore, the greater the position of the speaker to pronounce based on the unpronounced face, the higher the importance of the pronunciation of the vertex. In addition, even when the movement is not very large when pronounced, the vertices located near the large portion of the movement have a high importance for pronunciation. The expression of the emotions such as joy, sadness, surprise, fear, anger, etc. is also applied to the expressions, and the importance of the expression is determined. However, the mouth and its surroundings are a lot of movement not only for pronunciation but also for feelings. Like this, the part that moves a lot about pronunciation does not move less about expression. Therefore, it is necessary to observe the pronunciation and movement of expression at the same time to determine the importance.

When a person is not speaking, he expresses his emotions throughout his face, but when he is speaking, the parts that move for pronunciation, such as the mouth and surroundings, are subject to the movement of the pronunciation rather than the expression. Combining emotions into the lip-sync animations creates facial animations that express facial expressions as you speak and at the same time, consider the magnitude of movement for pronunciation in determining the importance of pronunciation and facial expressions for each vertex of the face model. Complement the movement of the expression according to its size. If the movement for the expression is small for a large part of the movement for pronunciation, the importance for pronunciation is further increased, and if the movement for the expression is large, the importance for the expression is reduced. In addition, when the movement for the expression is large for a small part of the movement for pronunciation, the importance for the pronunciation is further lowered, and in the case where the movement for the expression is small, the importance for the expression is increased. Based on this fact, the importance of final pronunciation and the importance of expression are calculated in three steps. In the first step, the importance is calculated according to the size of the movement. In the second phase, the importance is propagated around the critical peaks calculated earlier. In the final step, the final importance is determined by considering the importance of the pronunciation and facial expressions obtained above.

For each vertex Vi of the face model, the importance of pronunciation and facial expression, which are calculated in the first stage, are called primary pronunciation importance p1 (Vi) and primary facial expression importance e1 (Vi) , respectively. The importance calculated in the second stage is called secondary pronunciation importance p2 (Vi) and secondary expression importance e2 (Vi) , respectively. The importance calculated at the last stage is called final pronunciation importance p3 (Vi) and final facial expression importance e3 (Vi) , respectively. The primary pronunciation importance and primary facial expression importance for each vertex Vi are as follows.

에서 각 비즘 모델의 i번째 정점들까지의 거리들 중에 가장 큰 값에 따라 결정된다. 단, 중요도가 0과 1사이의 값을 갖도록 모든 정점에 대해서 가장 큰 값으로 나누어 준다. 일차 표정 중요도에 대해서도 마찬가지 방법으로 계산된다.The i-th vertex of the model's face is not the primary importance pronounce pronounce the i-th vertex of the non-face models

Is determined by the largest value of the distances to the i th vertices of each bismuth model. However, divide by the largest value for all vertices so that importance is between 0 and 1. The same applies to primary facial expression importance.

Secondary pronunciation importance and secondary facial expression importance for each vertex Vi are as follows.

However, Lp and Le are threshold values for determining adjacent points for expression and emotion, respectively, and S1 is a reference value for primary pronunciation importance. Lp , Le, S1 are user specified. As a result, each vertex is assigned the highest importance of the vertices higher than the given reference value S1 as the secondary pronunciation importance, otherwise the primary pronunciation importance is maintained. Similar calculations are made for the secondary pronunciation importance for each vertex. Considering the importance of secondary pronunciation and secondary facial expressions, the importance of final pronunciation and expression is defined as follows.

However, S2 is designated by the user as a reference value for secondary pronunciation importance. The reason for this definition is that the lower the importance of the second facial expression, the higher the final facial expression is important for the vertices with higher secondary pronunciation importance, and the higher the final facial expression is more important for the vertices with the lower secondary pronunciation importance.

11 shows the primary, secondary, and final pronunciation importance and facial expression importance. The brighter it is, the higher the importance. In order to combine the lip-sync animation and the facial expression expressing emotions, the final position of each vertex of the face model is determined in consideration of the final facial expression importance and the final pronunciation importance. Peaks of great importance for pronunciation should respond to movements on facial expressions while maintaining movements on pronunciation as much as possible. Similarly, vertices with less importance for pronunciation have a greater importance for facial expressions, and therefore must respond to movements with pronunciation while maintaining maximum movement with respect to facial expressions.

의 위치와 표정 애니메이션으로부터 얻어진 얼굴의 i번째 정점

의 위치, 그리고 i번째 정점의 최종 발음 중요도 p3(Vi)에 따라 발음과 표정이 결합된 얼굴의 i번째 정점 Vi의 위치가 결정된다. 무표정 모델의 i번째 정점

을 기준으로

의 위치를 나타내는 벡터

와

의 위치를 나타내는 벡터

는 공간 상에 한 평면을 결정한다. 따라서, i번째 정점의 최종위치

은 그 평면 위에 존재해야 한다. I th vertex of face obtained from lip sync animation

I th vertex of face obtained from the position and facial expression animation

The location, and i end of the first importance to pronounce p3 (Vi) and pronounce the i-th vertex Vi facial expressions combined according to the vertex position is determined. I th vertex of the expressionless model

based on this

A vector representing the location of

Wow

A vector representing the location of

Determines one plane in space. Thus, the final position of the i th vertex

Must be on that plane.

의 위치를 중요도의 크기에 따라 유지하도록 하여 발음에 대한 모습이 나타나게 하며

의 크기와 방향을 고려하여 도 12와 같이 표정에 대한 모습도 고려할 수 있다. 벡터 의 에 대한 수직성분과 평행성분을 각각 proj_{p ⊥}E 와 proj_{p ∥}E라고 하면,

의

에 대한 평행성분 proj_{p ∥}E가

에 영향을 주게 되면 그 정점이 발음을 하기 위해 이동하는 방향으로의 크기가 달라지기 때문에 발음 모습을 깨뜨리게 된다. 따라서,

와

의

에 대한 수직성분 proj_{p ⊥}E만을 고려하여 최종위치

을 결정한다. 발음에 대한 중요도가 낮고 표정 중요도가 높은 경우에는 반대로 적용하여 최종위치를 결정한다. 이상을 정리하면 얼굴 모델의 각 정점 Vi의 최종위치는 아래와 같다.Also, if the importance for pronunciation is high

Maintain the position of according to the magnitude of the importance so that the appearance of pronunciation appears

In consideration of the size and the direction of Figure 12 may also consider the appearance of the expression. vector of If the vertical and parallel components for are proj _{p ⊥} E and proj _{p ∥} E,

of

Parallel component proj _{p ∥E}

This affects the pronunciation because the size of the vertices changes in the direction of movement for pronunciation. therefore,

Wow

of

Final position considering only the vertical component proj _{p ⊥} E

Determine. In the case of low importance for pronunciation and high expression importance, the final position is determined by applying the opposite. In summary, the final position of each vertex Vi of the face model is as follows.

here,

to be. As a result, the vertices with great importance on pronunciation move in the direction of the vertical component of the movement on the expression, according to the magnitude of the importance, to reflect the movement on the pronunciation as much as possible and to save the movement on the expression. Even if the importance of pronunciation is small, the corresponding effect is obtained.

In order to form the above-described character, as shown in FIG. 13, the input unit 100, the lip sync animation forming unit 110, and the facial expression animation forming unit 120 may be provided. Then, the lip sync animation and the facial expression animation synchronizer 130 will be required, each function is as described above. In addition, the character data may be voice recording or voice data formed by the TTS system.

The present invention is not limited to the above-described embodiments, and such modifications are included within the scope of the present invention even though modifications may be made by those skilled in the art to which the present invention pertains.

Referring to the effects of the above-described character forming method and apparatus according to the present invention.

Firstly, it is possible to generate a lip sync animation representing the character's emotions in real time by obtaining input phoneme information from an existing TTS system or a voice file, and to represent emotions that continuously change with time according to a user input emotion interface that generates facial expression parameters. Can be.

Second, the bism models produced by the artist are arranged according to the phoneme information, and the relative positions of the bism models are adjusted according to the pronunciation length through Catmull-Rom spline interpolation to form a more natural and smooth lip sync animation.

Third, a lip-sync animation representing realistic emotion expressions can be produced by mixing facial expression models for various emotions with a lip-sync animation in an importance-based approach.

Fourth, because the animation production process according to the present invention is automated, it is possible to minimize the user's manual work consumed in producing the facial animation, and to maximize the accuracy of the voice, the naturalness and real-time of the animation.

Claims (22)

Receiving character data; Dividing a character's face model into a portion mainly used for pronunciation and a portion mainly used for expressing an expression, and animation of a portion mainly used for the pronunciation to form a lip sync animation of the character; And A step of forming a facial expression animation on the character on which the lip-sync animation is formed by animating a part mainly used to express the facial expression; Character forming method comprising a. The method of claim 1, Wherein said character data comprises voice information and facial expression information of said character. The method of claim 1, And the character data is input in the form of script text. The method of claim 3, wherein Character data in the form of the script text, character formation method characterized in that synthesized by the recorded training or TTS system. The method of claim 1, And synchronizing a lip sync animation formed on the character with an expression animation. The method of claim 1, The forming of the lip sync animation is performed by setting a phoneme sequence and a phoneme length of a character according to the character data. The method of claim 6, A character forming method comprising assuming that a mouth shape of a character is a key frame with respect to the set phoneme sequence, and processes each key frame by a keyframing animation technique. The method of claim 2, Divide the voice information into groups each having a similar mouth shape, Character input method characterized in that if the voice information in the same group is input, the lip sync animation of the character is formed in the same. The method of claim 8, The voice information is made in Korean, In the forming of the lip sync animation, the Korean language consisting of a total of 40 phonemes is divided into 11 vowel and consonant bismuth models. The method of claim 8, The voice information is made in English, The forming of the lip sync animation comprises: classifying English consisting of a total of 41 phonemes into 13 vowel and consonant bismuth models. The method of claim 1, wherein the formation of the lip sync animation, Character formation method comprising the Catmull-Rom spline interpolation method. delete delete The method of claim 1, wherein the facial expression animation, And a facial expression example model and a facial expression model representing emotions, respectively, and the facial expression example model and the expressionless model are mixed to express a change in emotion as a facial expression of a character. The method of claim 1, The forming of the lip sync animation and the facial expression animation is performed by an importance based approach. The method of claim 15, wherein the importance based approach comprises: And a lip sync animation or a facial expression animation for each of the points according to the importance of pronunciation and expression for each point of the mask of the character. The method of claim 1, wherein the forming of the facial expression animation, The character's face is divided into a forehead, eyes, eyebrows, nose and cheeks, and the forehead, eyes, eyebrows, nose and cheeks according to the input character data, characterized in that the character formation method. The method of claim 17, Lips of the face of the character, characterized in that the animation of the input character data, characterized in that the animation according to the material related to the lips. The method of claim 1, The character material divides the emotion of the character into any one of joy, surprise, sadness, fear and anger, The character expression animation is animated according to the emotion of the separated character. An input unit for inputting character data; A lip sync animation forming unit for classifying a face model of a character into a part mainly used for pronunciation and a part mainly used for expressing an expression, and animating a part mainly used for the pronunciation to form a lip sync animation; And Character expression apparatus comprising a facial expression animation forming unit for forming a facial expression animation on the character with the lip-sync animation is formed by animate the portion mainly used to represent the facial expression. The method of claim 20, Character forming apparatus further comprises a synchronization unit for synchronizing the lip sync animation and facial expression animation formed in the character. The method of claim 20, The input unit is a TTS system, characterized in that the character data is voice data.

Images