JP3298076B2 - Image creation device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[Table of Contents] The present invention will be described in the following order. Industrial application Conventional technology (FIG. 48) Problems to be solved by the invention (FIG. 48) Means for solving the problems (FIGS. 39 and 46) Action (FIGS. 39 and 46) Example (1) First Embodiment (1-1) Overall Configuration (FIGS. 1 and 2) (1-2) Mouth Shape Parameter (FIG. 3) (1-3) Weight Parameter (FIGS. 4 to 38) (1-4) Generation of weighting parameters (FIGS. 39 to 42) (1-5) Effects of embodiment (2) Second embodiment (2-1) Overall configuration (FIGS. 43 to 45) (2-2) Synthetic balance parameter (FIG. 46) (2-3) Effect of Second Embodiment (3) Other Embodiment (FIG. 47) Effect of the Invention

[0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image forming apparatus, and can be applied to, for example, an animation forming apparatus to which a computer graphics technique is applied.

[0003]

2. Description of the Related Art Conventionally, in this type of animation creating apparatus, a plurality of interpolation vectors are multiply interpolated using a weighted parameter time series, and the animation is generated using the interpolation vectors generated in this way. A proposal has been made to create one (Japanese Patent Application No. 2-256415).

That is, in the multiple interpolation method, an interpolation parameter vector p having m parameter elements is used.
For (t), a basic vector p0 having the basic value of each parameter element as an element is set, and the basic vector p0 is subtracted from n interpolation parameter vectors pi (i = 1... N) to obtain m * n interpolation parameter matrix P (t)
To form

In the multiple interpolation method, a weighted parameter vector w (t) having n parameter elements is set, and an interpolation parameter matrix P (t) and a basic vector p0 are set.
From the following equation

(Equation 1) A vector a (t) having m elements represented by is newly generated.

Thus, as shown in FIG. 48, in the animation creating apparatus, the motion of an object (hereinafter referred to as an object) to be subjected to one motion is subjected to multiple interpolation of a plurality of object shapes and a weight parameter time series. Generated.

That is, in the animation creating apparatus, a plurality of objects N1, N2, and N3 having the same definition as the object giving the motion are prepared, and different postures are given to each of them (hereinafter referred to as interpolation objects).

Further, in the animation creating apparatus, a basic object K having the basic posture of the interpolation object is given corresponding to the basic vector p0, thereby defining the interpolation objects N1, N2 and N3. And n
The basic vector p0 defining the basic object K is subtracted from the interpolation parameter vectors p (t), and weighted mixing is performed using weighted parameter vectors w1, w2, and w3.

Thus, in the animation creation device, if the time series vector w (t) representing the attitude of the object at each time is generated by executing the arithmetic processing of the equation (1), the attitude information of the interpolation object is obtained. It can be generated by using a multiple interpolation method using an interpolation parameter vector, whereby the animation A can be created based on the vector w (t).

As a result, in the animation creating apparatus to which the multiple interpolation method is applied, compared with the case where the key frame method, which is the most general method of computer graphic animation, is applied, the animation has a smaller data amount. And has the feature of creating highly expressive and expressive animations.

[0011]

By the way, in an animation in which a person appears, an animation in which a mouth is moved in accordance with a spoken word (hereinafter referred to as "lip sync animation") is indispensable.

However, the conventional Lip Sync animation is a simple one in which the mouth is repeatedly opened and closed while most of them are speaking. In order to bring the Lip Sync animation closer to the actual movement of the mouth, in the conventional animation creation device, it is necessary to set a precise object model and drive the parameters of the object model.

That is, the motion of the obtained animation depends on what parameters the object model has, and even if a realistic motion can be realized, the animation data is transferred to another object model. There are features that are difficult to apply.

For this reason, in the case of creating a Lip Sync animation, the conventional animation creation device has a problem that it is difficult to create it simply and richly, and even once it is created, it is difficult to create various objects. There was a problem that could not be applied. On the other hand, in the animation creation apparatus to which the multiple interpolation method is applied, since the motion is basically described as a time series of weighted parameters, if the weighted parameters can be automatically generated, the animation can be efficiently performed. Can be created.

The present invention has been made in view of the above points, and makes it possible to easily generate a rich expression lip sync animation independent of an object model by utilizing the advantages of the multiple interpolation animation. It is intended to propose an image creating apparatus capable of doing so.

[0016]

According to the present invention, there is provided an image creating apparatus for creating animation data by a multiple interpolation method, and a method for generating weighted parameter time series in an image creating method. , 16,
In steps 18 and 32, the interpolation parameter time series W0, W2, W4 of vowels and special sounds, and the interpolation parameter time series W
1, W3, and W5, and then generates the weighted parameter time series W based on the sounding timing based on the sounding information 12.
6, W7 is generated, and animation creating means 6, 42
Thus, multiple interpolation of the mouth shape is performed using the weighted parameter time series W6 and W7, and an animation U3 in which the mouth shape changes according to the pronunciation information 12 is created.

Further, in the present invention, the mouth shape set in advance includes a mouth shape that pronounces “a”, a mouth shape that pronounces “i”, and a mouth shape that pronounces “u”.

Further, in the present invention, the weighted parameter time series generation means 4 sets the interpolation parameter element data forming the interpolation parameter time series W0 to W5 for all vowels and consonants of the generated voice. I did it.

Further, in the present invention, the weighting parameter time series generating means 4 changes the weighting parameter so that the strength of the consonant appearing in the mouth shape changes according to the type of consonant and / or the frequency of silence based on the pronunciation information 12. Time series W6 and W7 are formed.

Further, in the present invention, the weighted parameter time series generating means 4, 16, 18, 32 include interpolation parameter time series W0, W2, W4 of vowels and special sounds and interpolation parameter time series W1, W1 of consonants. W3 and W5 are combined to generate weighted parameter time series W6 and W7.

Further, in the present invention, the weighting parameter time series generating means 4 converts the pronunciation information 12, the vowel pronunciation information and the consonant pronunciation information so that the ratio between the movement of the vowel and the movement of the consonant is maintained at an appropriate value. The weighted parameter time series W6 and W7 whose values change in a time series based on the time series are formed.

[0022]

According to the preset mouth shape, the interpolation parameter time series W0,
W2, W4, and consonant interpolation parameter time series W1, W
3 and W5, the weighting parameter time series W6 and W7 based on the sounding timing based on the sounding information 12
And the generated weighted parameter time series W6, W6
If the animation U3 whose mouth shape changes in accordance with the pronunciation information 12 is created by performing multiple interpolation of the mouth shape by using the method 7, the advantage of the multiple interpolation method animation can be utilized to create an object model. An easy-to-generate lip-sync animation that does not depend on the expression can be generated.

[0023]

BRIEF DESCRIPTION OF THE DRAWINGS FIG.

(1) First Embodiment (1-1) Overall Configuration In FIG. 1, reference numeral 1 denotes an overall configuration of an animation creating apparatus for generating a lip sync animation as a whole. Generates a Lip Sync animation that synchronizes with the speech created by the rule synthesis method.

That is, in this embodiment, the animation creating apparatus 1 is adapted to synthesize speech in the speech rule synthesizing section 2 using a complex cepstrum rule speech synthesis system. A speech is synthesized based on a pronunciation text that describes a symbol and accent position information.

Here, as shown in FIG. 2, the pronunciation information is formed of phoneme information and pronunciation timing information.
For example, it can be represented by Roman characters or other symbols, and the sounding timing information can indicate the time at which each phoneme is sounded by predetermined numerical data.

In the animation creation apparatus 1, a weighting parameter is generated from phonemic information by the weighting parameter generation unit 4 based on the sounding timing information. Further, in the animation creation apparatus 1, the MIMe animation generation unit 6 creates an animation from the mouth shape model based on the weighting parameters.

Thus, in the animation creating apparatus 1, a moving pattern is generated from four stationary patterns (interpolation, basic object) and three weighted parameter time series.

(1-2) Mouth Shape Parameter In this embodiment, the animation creating apparatus 1 is set in advance with three patterns of mouth shape objects as interpolation objects expressing the shape of the mouth.

Here, as shown in FIG. 3, the first mouth-shaped object is a mouth-shaped object in which the lower jaw is lowered and the mouth is opened to generate the sound of "A" (FIG. 3 (A)). The shape object is a mouth shape (FIG. 3 (B)) that generates a sound of “I” in which the teeth are peeled and the lips are stretched sideways (FIG. 3 (B)). The mouth shape (FIG. 3 (C)) which generates the voice of (1) is selected.

As a result, the mouth shape that generates the voice of “e” can be expressed by an intermediate mouth shape between the mouth shape that generates the voice of “a” and the mouth shape that generates the voice of “i”. The mouth shape that generates the voice of “O” can be expressed as an intermediate mouth shape between the mouth shape that generates the voice of “A” and the mouth shape that generates the voice of “U”. Is capable of freely selecting and creating a mouth-shaped object for a vowel.

Further, in this embodiment, the animation creation device 1 prepares a basic pattern in which the mouth is naturally closed as a basic mouth shape object (FIG. 3 (D)). All mouth shapes can be expressed. If three kinds of vowel mouth shapes are selected for the three interpolation objects, various mouth shapes can be intuitively grasped when these three interpolation objects are selected as coordinate axes for setting parameters. Therefore, the overall usability can be improved accordingly.

Thus, in this embodiment, "a",
By expressing a Japanese pronunciation shape using four patterns of shapes corresponding to "i" and "u" and a basic shape, it is possible to generate a lip sync animation synchronized with the speech created by the rule synthesis method. It has been done.

(1-3) Weight Parameters Here, Japanese sound elements are divided into vowels, consonants, and special sounds. In this embodiment, weight parameter elements are selected. Here, the weighting parameter is a three-dimensional vector that determines the interpolation ratio of three mouth-shaped patterns each represented by an interpolation object. In this embodiment, a time-series of weighting parameters is generated using the weighting parameter elements. ,
A series of animations is created using this time series.

In this case, the mouth shape in the pronunciation of the Japanese vowel can be represented by three mouth shape patterns each represented by an interpolation object, so that the mouth shape in the pronunciation of the consonant is also represented by the time series of the mouth shape pattern. To get. For this reason, in this embodiment, the animation creating apparatus 1 defines weighted parameter elements for vowels with a predetermined vector pattern, and expresses consonants by changing the vector pattern using the same vector pattern.

That is, as shown in FIG. 4, the weight parameters of the vowels are "A", "I", and "U" on the three-dimensional coordinate axis, and each of the mouth shapes is represented by one. It can be expressed as a point.

On the other hand, as shown in FIG. 5, a consonant can be shown as a locus with respect to a three-dimensional coordinate axis having a mouth shape of "A", "I", and "U". As a result, the mouth shape of a vowel can be defined by a predetermined vector pattern, and consonants can be expressed by a similar vector pattern.

On the other hand, for a special sound consisting of a long sound, a prompt sound, and a silent sound, the animation creating apparatus 1 does not set a fixed weighted parameter element, but instead generates a special weighted parameter time series when generating a weighted parameter time series. It is generated from the time series of weighted parameters before and after according to the context of the sound.

By the way, in order to create a realistic ripsing animation with the animation creating apparatus 1,
It is necessary to set each element data so that the mouth shape of each sound element can be expressed well. For this reason, in this embodiment, as shown in FIGS. 5 to 38, weighted parameter element data is assigned to each consonant so that a realistic ripsing animation can be created. Here, the horizontal axis represents time (m
s), and the vertical axis represents the weighting strength. Further, the symbol “＠” represents a silenced consonant, the symbol “q” represents “g” of a muddy sound, and the symbol “x” represents “n”.

Accordingly, FIG. 6 shows the weighted parameter element data when the sound represented by the phonetic symbol "k" is generated, and FIG. 7 shows the weighted parameter element data when the sound is muted. FIG. 8 shows weighted parameter element data when a voice represented by the phonetic symbol “s” is generated, and FIG. 9 shows weighted parameter element data when the sound is muted.
FIG. 10 shows the weighted parameter element data of the voice represented by the phonetic symbol “sh”, and FIG. 11 shows the weighted parameter element data when the sound is muted.

FIG. 12 shows the weighted parameter element data of the voice represented by the phonetic symbol "t", FIG. 13 shows the weighted parameter element data of the voice represented by the phonetic symbol "ch", and FIG. Represents the weighted parameter element data at the time of execution. Further, FIG. 15 shows the weighted parameter element data of the voice represented by the phonetic symbol "ts", FIG. 16 shows the weighted parameter element data when the voice is silenced, and FIG. FIG. 18 shows the weighted parameter element data of the voice represented by the phonetic symbol "h".

FIGS. 19 to 30 show the pronunciation symbols "f", "m", "y", "r", "w", "g",
FIGS. 31 to 38 show the weighted parameter element data of the voice represented by “q”, “z”, “j”, “d”, “b”, and “p”, respectively. ny "," h
y "," my "," ry "," gy "," by ",
This represents weighted parameter element data of the voice represented by “x”.

(1-4) Generation of Weight Parameter As shown in FIG. 39, the animation creating apparatus 1 uses the mouth shape pattern prepared in advance and the weight parameter element to generate the weight parameter time series based on the phonetic information. Automatically generate, thereby automatically generating a ripping animation.

That is, in the animation creating apparatus 1, the weighted parameter time series W1, W3, W5 (hereinafter referred to as consonant weighted parameter time series) describing the movement of the mouth of only the consonant, and the movement of the mouth of the vowel and the special sound are recorded. Generate two time series of weighting parameter time series W0, W2, W4 (hereinafter referred to as vowel weighting parameter time series) to be described, and interpolate each weighting parameter time series with "A", "I", "U" Two kinds of mouth movements U1 and U2 are generated by applying a multiple interpolation method to the object.

Subsequently, in the animation creating apparatus 1, the two kinds of mouth movements U1 and U2 are determined by applying a multiple interpolation method to a predetermined mixed weight parameter W6,
Mix well using W7 to produce the final ripping animation U3. Here, the animation creating apparatus 1 arranges the weighted parameter elements corresponding to each consonant discretely on the time axis according to the sounding timing, and then interpolates the end of each element and the start of the next element, Generate a consonant weighted parameter time series.

At this time, if the weighted parameter elements of the preceding and following consonants overlap on the time axis, the animation creation device 1
Generates a consonant weighted parameter time series so as to vary continuously by applying an interpolation technique to generate a consonant weighted parameter time series.

On the other hand, the animation creation device 1
After determining the weighting parameters for the sound generation timing of each sound, the weighting parameter elements are similarly arranged discretely on the time axis, and then the weighting parameter elements are set so as to sequentially fill the gap by interpolation processing. Thus, a vowel weighted parameter time series is generated.

At this time, if the sound element at a certain sound generation timing is a vowel, the animation creating apparatus 1 sets the weight parameter value at that time as the weight parameter element of the vowel, while If the sound element is a long sound, the weight parameter value at the immediately preceding sounding timing time is set to the weight parameter value at that time.

Further, in the animation creating apparatus 1, if the sound element at a certain sound generation timing is a prompt sound, the weighting parameter value at that time is set to a zero vector (that is, all three parameters have the value 0). On the other hand, when the sound element at a certain sound generation timing is silent, a value obtained by multiplying the weighted parameter value of the immediately preceding sound generation timing by a constant attenuation ratio is set as the weight parameter value at that time.

As a result, it is possible to generate in real time a lip sync animation synchronized with a voice created by applying the rule synthesis method.

By the way, when the mouth movement is expressed only by the vowel weighted parameter time series generated in this way, it can be expressed only as a movement that is uncomfortable with the actual movement.
Therefore, in this embodiment, two kinds of mouth movements U1
And U2 are mixed in a well-balanced manner using predetermined mixing weight parameters W6 and W7, thereby realizing a realistic and expressive movement.

For this purpose, it is important to set weighting parameters W6 and W7 (hereinafter referred to as mixing weighting parameter time series) for determining the mixing ratio. Here, the mixed weighting parameter time series is a two-dimensional vector that determines the mixing ratio of both vowels and consonants. In this embodiment, based on the sounding timing information, the vowels,
A mixed weighted parameter time series is generated such that the consonant attracts the other party, thereby smoothly expressing the movement of the vowel and the consonant.

At this time, in the animation creating apparatus 1, the mixed weighting parameter time series is switched and generated so that the strength of attracting the other party is switched according to the type of the sound element of the consonant. For example, "ma", "ぱ",
To express the difference between the mouth shape that pronounces consonants such as “ba” and the consonants that do not appear as the shape of a sticky mouth, such as “ka” and “ha”, that sound these consonants It has been done.

As a result, the animation creating apparatus 1 was able to generate a realistic and expressive Ripping animation U3. Further, by using a mixture of two movements, an interaction due to the context of vowels and consonants can be expressed.

Specifically, when "ku" is pronounced as a single sound, the movement of the mouth is more pronounced than "u", whereas "ku" in a series of sounds produces a vowel. There was a characteristic that the movement of the mouth did not appear suddenly, and such a facial expression could be expressed.

Further, as shown in FIG. 40 and FIG. 41, it was confirmed that the mouth shape changes depending on the speaking speed as if a person actually spoke. In this case, the difference between "aiueo" and "kakikukeko" is due to the consonant [k], but it can be seen that the appearance of [k] differs between when talking loosely and when talking quickly.

Further, as shown in FIG.
By mixing the kinds of mouth movements U1 and U2 with a predetermined mixing weight parameter, this balance can be adjusted to create a characteristic speech. That is, FIG. 42 shows a case where the mixture ratio of the vowel mouth movement U2 and the consonant mouth movement U1 is varied, and the mixture ratio of the vowel increases toward the right side.

This makes it possible to automatically generate a weighted parameter time series so as to reflect the actual mouth movement, thereby real-time processing using a pre-set mouth shape pattern by applying the multiple interpolation method. Can automatically create animations. Furthermore, at this time, it is possible to easily create a lip-sync animation simply by inputting Japanese pronunciation information (that is, consisting of sound elements and timing), and furthermore, it is possible to express a difference in mouth shape by each consonant, thereby realizing a realistic sound. You can create animations.

In combination with the speech synthesis system, an animation for reading Japanese text information can be automatically created. Further, the same motion data can be shared by different mouths (object models), thereby making it possible to make a library of motion data.

That is, the weighting parameter of the multiple interpolation animation is characterized in that it does not depend on the object model. Therefore, in a normal animation creation device,
For example, for an object composed of several representative points, the movement of this object is expressed as a time series of the movement of each representative point, and at this time, the number of representative points of the target object is changed and the original motion data is applied Even in such a case, in the multiple interpolation animation, even if the number of representative points is changed, the original motion data can be applied, so that the animation data can be made into a library.

Therefore, according to this embodiment, the weighted parameter time series as motion data can be used in common for various objects. Also, if the setting of the interpolation object model is changed, it is possible to move objects other than the mouth in synchronization with words using the same motion data.

(1-5) Effects of the Embodiment According to the above configuration, when creating a mouth-shaped animation by applying the multiple interpolation method, a weighted parameter time series is generated based on the timing of sound generation. By doing so, it is possible to easily generate an expressive Lip Sync animation that does not depend on an object model that can move the mouth in accordance with the words spoken.

(2) Second Embodiment (2-1) Overall Configuration In FIG. 43, reference numeral 10 denotes an animation creation device according to the second embodiment as a whole, and the pronunciation information is the same as in the first embodiment. To automatically generate weighted parameter time series.

In the animation creating apparatus 10,
The pronunciation information 12 is provided to a pronunciation information separation unit 14, where text processing is performed to separate the vowel and consonant pronunciation information, and output to the motion information generation units 16 and 18 as vowel pronunciation information and consonant pronunciation information, respectively. .

The motion information generators 16 and 18 generate a parameter time series representing a change in mouth shape based on vowel pronunciation information and consonant pronunciation information, respectively. That is, as shown in FIG. 44, the motion information generating units 16 and 18 transmit the pronunciation information including the vowel pronunciation information and the consonant pronunciation information to the pronunciation information input unit 2.
0, and the sound information is stored in the sound information memory 22.

The motion information generators 16 and 18 store preset mouth shape parameters in the mouth shape data memory 24.
The mouth shape parameters stored in the mouth shape data memory 24 are sequentially selected based on the sound information.

Further, the arithmetic unit 26 stores the selected mouth shape parameter in the motion information time-series memory 28, and at this time, selects a predetermined address in accordance with the sounding timing obtained from the sounding timing information to change the mouth shape parameter. Store. On the other hand, the motion information output unit 30 reads and outputs the contents of the motion information time series memory 28 in address order, and thereby outputs the motion information time series consisting of the time series of the mouth shape parameters to the motion information synthesis unit 32. .

Thus, the motion information generators 16 and 18
Outputs a motion information time series composed of a time series of mouth shape parameters as vowel motion information and consonant motion information by applying the same method as in the first embodiment.

The motion information synthesizing section 32 generates one motion information using the vowel motion information, the consonant motion information, and the synthesis balance parameter. That is, as shown in FIG. 45, the motion information synthesizing unit 32 inputs the vowel motion information and the consonant motion information via the motion information input unit 34, but inputs the synthetic balance parameter via the synthetic balance parameter input unit 36. Then, arithmetic processing is sequentially performed by the arithmetic unit 38.

At this time, in the arithmetic unit 38, motion information composed of time-series data is generated by successively performing multiple interpolation arithmetic processing at predetermined time intervals, and this motion information is transmitted via the motion information output unit 40. And outputs it to the image generation device 42. Here, the image generation device 42 creates a ripple animation based on the motion information and displays it on the monitor device 44.

Here, the synthetic balance parameter is a weighting parameter for interpolating two mouth shape parameters consisting of a vowel and a consonant. In this embodiment, by using this synthetic balance parameter, a rich expression of Ritz can be obtained. Create a pushing animation.

(2-2) Synthetic Balance Parameter As shown in FIG. 46, the synthetic balance parameter is generated by the synthetic balance parameter generator 46 based on the pronunciation information 12, vowel sound information, and consonant sound information. The vowel motion information and the consonant motion information have the same relationship as the mixed weight parameters W6 and W7 (FIG. 39) of the first embodiment.

In this embodiment, the combined balance parameters are converted to combined balance parameter time series W7, W8.
To maintain the ratio of vowel movements to consonant movements in an appropriate relationship, thereby creating a richly expressive ripsing animation.

That is, the synthesis balance parameter generation unit 4
In 6, the value becomes maximum at each sounding timing,
The synthesis balance parameter is generated such that the value gradually decreases as the distance from the timing of the maximum value increases. Thereby, in the animation creating apparatus 10, it is possible to prevent a movement in which the movement of the vowel and the movement of the consonant interfere with each other's movement, and create a realistic ripping animation. .

Further, the synthesis balance parameter generating section 46
In, the time interval of each phoneme is detected by sequentially detecting the sounding timing, and thereby the sounding speed information is detected.

Further, the synthesis balance parameter generating section 46
In, based on this speed information, when the pronunciation speed is high, the synthetic balance parameter is generated such that the movement of either the vowel or the consonant is reduced, or both movements are reduced. As a result, in the animation creating apparatus 10, it is possible to create a realistic ripping animation that does not lose its smoothness even when the speed of sound generation is high.

Further, the synthesis balance parameter generating section 46
In, the information unique to the phoneme is stored in a predetermined memory circuit, whereby the importance expressed in the mouth shape of the consonant is stored as the weight information value according to the phoneme of the consonant. As a result, the animation creation device 10 can correct the synthesis balance parameter based on this value, and thus can create a more realistic ripple animation.

(2-3) Effects of the Second Embodiment According to the configuration of FIG. 43, a synthetic balance parameter is generated based on pronunciation information, vowel pronunciation information, and consonant pronunciation information, whereby the movement of vowels and consonants are generated. By maintaining the ratio of the movement to the appropriate relationship, it is possible to create a rich expression animation.

(3) Other Embodiments In the above-described second embodiment, the case where the motion information time series is output from the motion information generation unit as the time series of the mouth shape parameter has been described. Not only
As shown in FIG. 47, a weighting parameter element memory 48 for storing weighting parameter elements and a weighting parameter time series data memory 50 for storing weighting parameter time series generated by the weighting parameter elements are added to the motion information generation unit.
Thus, the weight information time series may be directly output from the motion information output unit 30.

Further, in the above-described embodiment, the case where the pronunciation information is input from the speech synthesis system has been described. However, the present invention is not limited to this. For example, the pronunciation information is detected from the real voice, and the pronunciation information is used. You may.

Further, in the above-described embodiment, the case where the weighting parameter is switched between the case where the user speaks slowly and the case where the user speaks quickly is described. However, the present invention is not limited to this. May be used to switch the weighting parameter. That is, the amplitude of the vowel-weighted parameter time series is related to the size of the opening of the mouth. In contrast, the amplitude of the consonant weighted parameter time series is related to the loudness of the mouth movement. Therefore, for example, by detecting information such as the volume of the voice and the tone of the voice, and modulating the mixing weight parameter based on the detection result,
Animations closer to audio images can be generated.

[0082]

As described above, according to the present invention, the interpolation parameter time series of vowels and special sounds and the interpolation parameter time series of consonants are determined based on pronunciation information for a preset mouth shape. After generation, a weighted parameter time series is generated based on the sounding timing based on the pronunciation information, and multiple interpolation of the mouth shape is performed by using the generated weighted parameter time series, so that the mouth shape is determined according to the pronunciation information. An image creation apparatus that can easily generate rich lip sync animations that do not depend on an object model by taking advantage of the multiple interpolation animation method by creating an animation whose shape changes. An image creation method can be realized.

[Brief description of the drawings]

FIG. 1 is a schematic diagram illustrating an overall configuration of an animation creating apparatus according to an embodiment of the present invention.

FIG. 2 is a schematic diagram illustrating pronunciation information.

FIG. 3 is a schematic diagram illustrating a mouth-shaped object.

FIG. 4 is a characteristic curve diagram showing weight parameter elements of vowels.

FIG. 5 is a characteristic curve diagram showing weight parameter elements of consonants.

FIG. 6 is a characteristic curve diagram showing weighted parameter element data of a voice represented by a phonetic symbol “k”.

FIG. 7 is a characteristic curve diagram showing weighted parameter element data at the time of silence.

FIG. 8 is a characteristic curve diagram illustrating weighted parameter element data of a voice represented by a phonetic symbol “s”.

FIG. 9 is a characteristic curve diagram showing weighted parameter element data at the time of silence.

FIG. 10 is a characteristic curve diagram showing weighted parameter element data of a voice represented by a phonetic symbol “sh”.

FIG. 11 is a characteristic curve diagram showing weighted parameter element data at the time of silence.

FIG. 12 is a characteristic curve diagram illustrating weighted parameter element data of a voice represented by a phonetic symbol “t”.

FIG. 13 is a characteristic curve diagram showing weight parameter element data of a voice represented by a phonetic symbol “ch”.

FIG. 14 is a characteristic curve diagram showing weighted parameter element data at the time of silence.

FIG. 15 is a characteristic curve diagram showing weighted parameter element data of a voice represented by a phonetic symbol “ts”.

FIG. 16 is a characteristic curve diagram showing weighted parameter element data when the sound is muted.

FIG. 17 is a characteristic curve diagram illustrating weighted parameter element data of a voice represented by a phonetic symbol “n”.

FIG. 18 is a characteristic curve diagram illustrating weighted parameter element data of a voice represented by a phonetic symbol “h”.

FIG. 19 is a characteristic curve diagram illustrating weighted parameter element data of a voice represented by a phonetic symbol “f”.

FIG. 20 is a characteristic curve diagram illustrating weighted parameter element data of a voice represented by a phonetic symbol “m”.

FIG. 21 is a characteristic curve diagram showing weight parameter element data of a voice represented by a phonetic symbol “y”.

FIG. 22 is a characteristic curve diagram showing weight parameter element data of a voice represented by a phonetic symbol “r”.

FIG. 23 is a characteristic curve diagram showing weight parameter element data of a voice represented by a phonetic symbol “w”.

FIG. 24 is a characteristic curve diagram showing weighted parameter element data of a voice represented by a phonetic symbol “g”.

FIG. 25 is a characteristic curve diagram illustrating weighted parameter element data of a voice represented by a phonetic symbol “q”.

FIG. 26 is a characteristic curve diagram illustrating weighted parameter element data of a voice represented by a phonetic symbol “z”.

FIG. 27 is a characteristic curve diagram showing weight parameter element data of a voice represented by a phonetic symbol “j”.

FIG. 28 is a characteristic curve diagram illustrating weighted parameter element data of a voice represented by a phonetic symbol “d”.

FIG. 29 is a characteristic curve diagram illustrating weighted parameter element data of a voice represented by a phonetic symbol “b”.

FIG. 30 is a characteristic curve diagram showing weight parameter element data of a voice represented by a phonetic symbol “p”.

FIG. 31 is a characteristic curve diagram showing weighted parameter element data of a voice represented by a phonetic symbol “ky”.

FIG. 32 is a characteristic curve diagram showing weight parameter element data of a voice represented by a phonetic symbol “ny”.

FIG. 33 is a characteristic curve diagram showing weighted parameter element data of a speech represented by a phonetic symbol “hy”.

FIG. 34 is a characteristic curve diagram showing weighted parameter element data of a voice represented by a phonetic symbol “my”.

FIG. 35 is a characteristic curve diagram showing weighted parameter element data of a voice represented by a phonetic symbol “ry”.

FIG. 36 is a characteristic curve diagram showing weighted parameter element data of a voice represented by a phonetic symbol “gy”.

FIG. 37 is a characteristic curve diagram showing weighted parameter element data of a voice represented by a phonetic symbol “by”.

FIG. 38 is a characteristic curve diagram showing weight parameter element data of a voice represented by a phonetic symbol “x”.

FIG. 39 is a schematic diagram for explaining automatic generation of a ripping animation.

FIG. 40 is a characteristic curve diagram showing the movement of the mouth when the user speaks loosely.

FIG. 41 is a characteristic curve diagram showing mouth movements when speaking quickly.

FIG. 42 is a schematic diagram showing actual mouth movements.

FIG. 43 is a block diagram showing an animation creating apparatus according to a second embodiment.

FIG. 44 is a block diagram showing a motion information generation unit.

FIG. 45 is a block diagram illustrating a motion information synthesis unit.

FIG. 46 is a schematic diagram used for describing a synthesis balance parameter;

FIG. 47 is a block diagram showing another embodiment of the motion information generator.

FIG. 48 is a schematic diagram for explaining creation of animation using the multiple interpolation method;

[Explanation of symbols]

1, 10... Animation creation device, 2... Voice rule synthesizing unit, 4... Weighting parameter generating unit, 6... MIMe
Animation generation unit, 12 ... pronunciation information, 14 ... pronunciation information separation unit, 16, 18 ... motion information generation unit, 32 ...
... Motion information synthesis unit.

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-4-133182 (JP, A) JP-A-2-16681 (JP, A) JP-A-63-225875 (JP, A) Supports "text information" Synthesizing a Face Moving Image with Shape Change ”, IEICE Transactions, Vol. J75-DII, No. 2, p. 203-215 (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T 13/00 G06T 11/60 H04N 5/262 CSDB (Japan Patent Office)

Claims (12)

(57) [Claims] An image creating apparatus for creating animation data by a multiple interpolation method, comprising: a vowel based on pronunciation information for a preset mouth shape;
And interpolation parameters of special sounds and the consonant interpolation parameters
After generating a meter time series, a weighted parameter time series generating means for generating a weighted parameter time series based on the pronunciation timing based on the pronunciation information, and the mouth shape using the generated weighted parameter time series An image creating apparatus, comprising: animation creating means for creating an animation whose mouth shape changes according to the pronunciation information by performing multiple interpolation. 2. The method according to claim 1, wherein the preset mouth shape is a mouth shape that pronounces “A”, a mouth shape that pronounces “I”, and a mouth shape that pronounces “U”. The image creation device described in the above. 3. The weighted parameter time series generating means includes: an interpolation parameter time series of the vowel and special sound;
The weighted parameter is synthesized by synthesizing the sound interpolation parameter time series.
The image creation device according to claim 1 , wherein the image creation device generates a data time series . 4. The weighting parameter time series generating means according to claim 1 , further comprising:
Element data of interpolation parameters that generate parameter time series
Image creating apparatus according to claim 3, characterized in that to set the. 5. The weighted parameter time series generating means according to the type of consonant and / or the frequency of silence based on the pronunciation information.
In the same way, the intensity of the consonant appearing in the mouth shape changes,
The image creating apparatus according to claim 3 , wherein the weighted parameter time series is generated . 6. The weighted parameter time series generating means holds an appropriate ratio between a vowel movement and a consonant movement.
Based on pronunciation information, vowel pronunciation information, and consonant pronunciation information.
Weighting parameter such that the value changes in time series
The image creation device according to claim 3 , wherein the image creation device generates a time series . 7.Animation data by multiple interpolation
In the image creation method to create, Vowel based on pronunciation information for preset mouth shape
And interpolation parameters of special sounds and the consonant interpolation parameters
A first step of generating a meter time series; Weighted based on the pronunciation timing based on the above pronunciation information
A second step of generating a parameter time series; The mouth shape using the generated weighted parameter time series
Multi-interpolation of the shape of the mouth
The third step of creating animated animations With
An image creation method characterized in that it is obtained. 8. The preset mouth shape pronounces "a"
Mouth shape to pronounce "I" and mouth shape to pronounce "U"
Image <br/> image creating method according to claim 7, characterized that you become the mouth shape. 9.In the second step, The interpolation parameter time series of the vowels and special sounds and the consonants
The weighted parameter is synthesized by synthesizing the sound interpolation parameter time series.
Generate data time series Claims characterized by the following:7Described in
Image creation method. The method according to claim 10, wherein said first stearyl class tap, for all vowels and consonants of the generated sound, the inner挿Pa
Element data of interpolation parameters that generate parameter time series
Image generating method according to claim 9, characterized in that to set the. The method according to claim 11 wherein said second stearyl class tap, response to the type and or silence frequency consonant by the phonetic information
In the same way, the intensity of the consonant appearing in the mouth shape changes,
The method according to claim 9 , wherein a weighted parameter time series is generated . The method according to claim 12 wherein said second stearyl class tap, to hold the ratio between the movement of the movement and consonant vowel to appropriate values
Based on pronunciation information, vowel pronunciation information, and consonant pronunciation information.
Weighting parameters such that the values change over time
The method according to claim 9 , wherein a time series is generated .