JP2007299300A - Animation creating device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an animation creating device capable of creating a moving image in which a portion of a shape is properly changed in accordance with voice without depending on a speaker. <P>SOLUTION: An animation creation system 80 includes a key frame data creation unit 96 for receiving a voice signal and creating key frame data representing a key frame image comprising face images at a prescribed key frame time during the continuous time of each phoneme in a phonemic sequence representing the voice signal, and an animation reproduction unit 98 for generating an animation composed of a series of images that changes in synchronization with the voice signal on the basis of the key frame data created by the key frame data creation unit 96. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an animation creating apparatus that creates an animation from voice, and more particularly to an apparatus that automatically generates an animation such as a face image that changes the shape of a mouth or the like in accordance with an uttered voice.

  Due to the development of computer technology, work that has been mostly done manually has been replaced by computer work. A typical example is the creation of animation.

  Previously, animations were generally created by the following method. The animation director decides the characters to appear, and creates a rough original picture of the main scene called a storyboard. Based on these storyboards, a picture of each frame of animation is created by an operator called an animator. The person in charge finishes these pictures into cel drawings. If the cell images are sequentially copied onto the film and played back at a predetermined frame rate, an animation image portion is completed.

  While playing this animation image, the voice actor adds a line based on the script of the animation. This is so-called “post-recording”.

  It is the creation of cel images that requires the most work in such work. On the other hand, when creating an original picture with CG (computer graphics), it is a relatively simple task to create a cell picture by processing the original picture. There is no need to shoot one by one. Therefore, this part is considerably computerized together with the CG conversion of the original picture.

  On the other hand, the remaining task is relatively difficult. Because it is necessary to speak dialogue with the voice of the situation according to the movement of the animation, the post-recording work takes a certain amount of time, and learning is also necessary.

  Therefore, conversely to post-recording, a method of recording audio first and creating an animation according to the audio was considered. This is called "Presco" or "Pre-Reco" (hereinafter referred to as "Presco etc."). This is a technique that was originally used when creating animations manually in the United States. When an animation is created by this method, the work procedure is as follows.

  First, determine the character that will appear in the animation. Create storyboards as before. A voice actor speaks based on a storyboard and script and records it as audio. An animation is created according to this sound.

  When the animation creation by the technique such as Presco is realized by a computer, the problem is how to automatically create the animation from the voice. In particular, it is difficult to generate the movement of the mouth of an animation such as a person in a natural form in accordance with the voice of a voice actor that has been recorded in advance, and there is a need for a method that automatically performs this.

One method proposed for this purpose is the method described in Patent Document 1. In the method described in Patent Document 1, a plurality of mouth-shaped basic patterns are prepared in advance. Then, a mouth shape corresponding to an arbitrary voice is obtained by a weighted sum of these basic patterns. For this purpose, a conversion function for converting a predetermined feature amount of the voice actor's voice into a weighting parameter of each basic pattern is obtained in advance by multiple regression analysis. The voice actor's voice is recorded into the voice actor's voice by converting the predetermined feature quantity of the voice actor's voice recorded along the script into a weighted parameter using this conversion function and calculating the weighted sum of the basic pattern of the mouth shape using the weighted parameter. Corresponding mouth shape and face image are created. By performing such a process at a time corresponding to each frame of the animation, an animation frame sequence is created.
Japanese Patent Laid-Open No. 7-44727

  In modern times, for example, the amount of communication involving moving images, such as remote conferences and videophones, is increasing. Therefore, how to reduce the amount of moving image data is a problem. One measure for this is to transmit only voice in communication, but to synthesize a face image from the voice on the receiving side. In order to generalize such a technique, a technique for appropriately generating a mouth image corresponding to an unspecified number of human voices is necessary.

  Further, as a service using the above-described animation creation, for example, a service that creates an animation using a face image of a specific character in accordance with the voices of an unspecified number of speakers can be considered. In such services, it is necessary to generate mouth movements of facial images appropriately from the voices of unspecified speakers.

  However, in the technique disclosed in Patent Document 1 described above, it is necessary to obtain a conversion function in advance. Therefore, even if it is effective for a specific speaker, it cannot be applied to an unspecified number of speakers. This is because, depending on the speaker, even if the phoneme uttered is the same, there are various acoustic feature quantities obtained from the speech.

  SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide an animation creating apparatus that can generate a moving image in which a part of the shape is appropriately changed according to a voice without depending on a speaker.

  The animation creating apparatus according to the first aspect of the present invention receives a voice signal, and a key frame image composed of images at a predetermined key frame time during the duration of each phoneme in the phoneme string represented by the voice signal. An animation for generating an animation of an image made up of a series of images that change in synchronization with an audio signal based on the key frame data that is generated by the key frame data generation means Generating means.

  The key frame data creation means receives the audio signal and creates key frame data representing a key frame image constituted by an image at a predetermined key frame time during the duration of each phoneme in the phoneme string represented by the audio signal. To do. The animation generation unit generates an animation of an image including a series of images that change in synchronization with the audio signal, based on the key frame data generated by the key frame data generation unit. Key frame data is created from the audio signal, and an image animation is generated from the key frame data. The key frame data is determined without depending on the speaker of the audio signal. Therefore, it is possible to provide an animation creation device that can appropriately generate a moving image from voice without depending on a speaker.

  Preferably, the predetermined key frame time is the start time of the duration of each phoneme in the phoneme string.

  The characteristics of the mouth shape when a phoneme is pronounced are most apparent when it is the first time to pronounce the phoneme. Therefore, when the predetermined key frame time is set as the start time of the phoneme duration, the obtained moving image becomes an appropriate one that well reflects the change in sound.

  More preferably, the animation creating device records the user's voice based on the selection of the text by the text selecting means for allowing the user to select a plurality of predetermined types of text and the text selecting means. Means for converting to a signal and providing it to the key frame data creation means along with the selected text. Means for creating key frame data includes mapping data storage means for storing mapping data for mapping a phoneme to any one of a plurality of predetermined images including a predetermined reference image, audio signal and text Phoneme segmentation means for performing phoneme segmentation on the speech signal based on the received text and outputting the phoneme string data including the obtained phoneme string and time information indicating the duration of each phoneme, and phoneme segmentation means For each phoneme included in the phoneme string data output from the phoneme, by referring to the time information of the phoneme and mapping data, an identifier for specifying an image to which the phoneme is mapped, and a predetermined feature for the phoneme Keyframe data by adding a blend ratio determined according to the volume. The create and includes a key frame data creation means for outputting.

  Phoneme segmentation is performed based on the text selected by the text selection means. Since each phoneme constituting the speech signal is known in advance, phoneme segmentation can be performed correctly.

  More preferably, the key frame data creation means includes, for each phoneme included in the phoneme string data output from the phoneme segmentation means, an image identifier obtained by referring to the mapping data, and a blend rate composed of a predetermined constant. And a mapping processing means for outputting image-mapped phoneme string data, and for each phoneme of the image-mapped phoneme string data output by the mapping process means as a monotonically increasing function of the duration of the phoneme And a first blend rate adjusting means for adjusting the blend rate.

  The phoneme continuation length reflects the rate of change in the shape of the mouth or the like when the phoneme is pronounced. Therefore, by adjusting the blend rate as a monotonically increasing function with respect to the phoneme duration, an animation that appropriately reflects the actual change in the shape of the mouth or the like can be obtained.

  The key frame data creation means further outputs, for each phoneme of the phoneme string data adjusted by the blend ratio, outputted from the first blend ratio adjustment means, as a monotonically increasing function of the magnitude of the power within the duration of the phoneme. Second blend rate adjusting means for adjusting the blend rate may be included.

  The power of the phoneme during the duration reflects the strength at which the phoneme is pronounced, and thus the rate of change in the shape of the mouth and the like at that time. Therefore, by adjusting the blend rate as a monotonically increasing function with respect to the power during the phoneme duration, it is possible to obtain an animation that appropriately reflects the actual change in the shape of the mouth or the like.

  Preferably, the animation generation means includes a time determination means for determining the generation time for generating the animation image in relation to the recording time of the sound, and an image of the frame at the generation time determined by the time determination means. Interpolating means for calculating by interpolating between images of a plurality of key frames sandwiching the generation time.

  Interpolation means generates an image of a frame at a certain generation time by interpolation between a plurality of key frame images including that time. The shape of the mouth or the like at a certain time is a shape in the middle of the transition from the previous phoneme to the next phoneme. Thus, by calculating the shape of the mouth or the like at a certain generation time by interpolation, an appropriate image animation corresponding to the phoneme transition can be created.

  More preferably, the interpolation means calculates means for calculating an image of the frame at the generation time determined by the time determination means by interpolation between two key frame images adjacent to each other across the generation time. Including.

  Interpolation is performed between two adjacent key frames across the generation time, and an image of the frame at the generation time is generated. While reducing the amount of calculation, appropriate interpolation can be performed and a smoothly changing animation can be obtained.

  More preferably, the means for calculating is 100% in the first key frame and 0% in the second key frame among the first and second key frames adjacent to each other across the generation time. First interpolation means for interpolating the first blend rate at the generation time from the blend rate in the first key frame by the first interpolation function, and 0% in the first key frame, the second key frame The second interpolation function for interpolating the second blend rate at the generation time from the blend rate in the second key frame by the second interpolation function that is 100% in the first key frame, the first blend rate and the second The weighted sum between the image data mapped to the first keyframe and the image data mapped to the second keyframe using the blend ratio. Ri, and means for calculating the data of an image in the generation time.

  The blend rate at the first key frame and the blend rate at the second key frame were calculated separately by the first and second interpolation functions, and then used to map to the first key frame. A weighted sum between the image data and the image data mapped to the second key frame is calculated. By combining simple calculations, a smooth animation of the image between two key frames can be calculated.

  The time determination means may include means for determining a time when an image of a certain frame is obtained by the interpolation means as a generation time for generating an image of the next frame.

  When the generation of the image by the interpolation means is completed, the time is determined as the generation time. When the generation time is determined, an image of a frame at the generation time is calculated by an interpolation unit by interpolation between a plurality of key frame images sandwiching the generation time. Therefore, the interpolating means always operates to generate an image without taking a rest, and the interpolating means can be used effectively.

  The image may be a face image reflecting a change in the shape of the mouth at the time of speech.

  When the computer program according to the second aspect of the present invention is executed by a computer, it causes the computer to function as each means constituting any one of the animation creating apparatuses described above.

  Hereinafter, the present invention will be described based on embodiments. In the following description, six types of basic face images are used, but the number of face images is not limited to this. There may be fewer than six types or more than six types.

[First Embodiment]
<Configuration>
FIG. 1 shows an outline of an animation creation process 30 by the animation creation apparatus according to the first embodiment of the present invention. Referring to FIG. 1, in the animation creation process 30, when a speaker 40 utters a speech based on a script 44, a phoneme segmentation (a phoneme sequence that constitutes an utterance from the speech) is performed on the speech signal 42. Is generated).

  For main phonemes, face images 60 to 68 including a mouth shape when the phoneme is pronounced are prepared in advance, and these face images are assigned to each phoneme 50 to 58 obtained as a result of speech recognition. To animate.

  Note that even if one speech image is assigned to each phoneme, a smooth image cannot be obtained. Therefore, in this embodiment, as described later, “A (/ a /)” is used as a main face image. Five face images for five phonemes of “I (/ i /)”, “U (/ u /)”, “E (/ e /)”, “O (/ o /)”, and expressionless face images, A total of six face images are prepared. The five phonemes “A” to “O” are assigned to the corresponding face images, respectively, and the remaining phonemes are assigned to any of the six face images described above. This is hereinafter referred to as phoneme to face image mapping.

  Furthermore, if face images mapped in this way are assigned to each phoneme and are simply connected to create an animation, the movement of the image becomes excessive, resulting in a so-called “noisy” animation. Therefore, in this embodiment, the “strength” of each image is adjusted according to the duration time of the phoneme and its power, and a face image in the transition process between phonemes is generated by interpolation using the adjusted image. . In addition, for a phoneme having a duration or power smaller than a predetermined threshold value, an image corresponding to the phoneme is not inserted, and the phoneme image immediately before that is integrated. By doing so, it is possible to generate a natural animation that changes smoothly according to the sound.

  FIG. 2 shows a schematic functional configuration of the animation generation system 80 according to the present embodiment. This animation generation system 80 prepares a plurality of transcription texts in advance, causes a speaker to select one of them and utters it, and animates a facial image that changes in accordance with the uttered speech in advance. It is generated by interpolation from the prepared six face images.

  Referring to FIG. 2, an animation generation system 80 includes a text selection interface 90 used when a speaker writes and selects text, a microphone 92 that converts a speaker's voice into an audio signal, and a plurality of types. Is stored, and one of them is selected by the text selection interface 90, and then the voice signal output from the microphone 92 is recorded to create a digitized voice data file. The phoneme segmentation for the input instruction unit 94 and the voice data file given from the input instruction unit 94 is performed using the corresponding transcription text given from the input instruction unit 94, and the result and the voice from the input instruction unit 94 are recorded. Based on the data file, And a key frame data creation unit 96 for creating a key frame data defining a beam.

  The animation generation system 80 further uses the audio data file output from the input instruction unit 94 and the key frame data output from the key frame data creation unit 96 to change the mouth shape in synchronization with the audio of the audio data file. An animation reproduction unit 98 for creating an animation of a face image and outputting it together with sound, and a monitor 102 for displaying animation and a speaker 100 for reproducing sound, both connected to the animation reproduction unit 98. Including.

  The input instruction unit 94 presents a text storage unit 110 for storing a plurality of kinds of transcription texts in advance, and presents the text stored in the text storage unit 110 to the speaker 40 through the text selection interface 90. A text selection unit 112 for selecting one of them and giving the text to the key frame data generation unit 96 and giving an instruction to the speaker to speak the text using the text selection interface 90; The voice signal output from the microphone 92 for the voice of the text uttered in response to the instruction of the text selection unit 112 by the speaker is framed with a predetermined frame length and frame shift length and stored as voice data to create key frame data Unit 96 and animation playback unit 98 And a voice recording unit 114 for giving.

  The key frame data creation unit 96 performs phoneme segmentation on the speech data from the speech recording unit 114 based on the text given from the text selection unit 112, and outputs phoneme sequence data including the phoneme sequence and its duration length. Speech recognition apparatus 120 for mapping, mapping table storage unit 130 storing a mapping table for mapping all phonemes constituting Japanese to the identifiers of the six face images described above, and phonemes output from the speech recognition apparatus 120 Parameters for creating a face image of a frame at a main time point in an animation from the above-described six face images based on the sequence file, the text given from the text selection unit 112, and the voice data given from the voice recording unit 114 And output as key frame data And a key frame data creation unit 136 for.

  The speech recognition device 120 may be any device as long as it can perform phoneme segmentation and output a phoneme string and time data for which each duration is known. Since the utterance content is known in advance, the speech recognition apparatus 120 can reliably convert speech data into a phoneme string.

  FIG. 5 shows a configuration example of the phoneme string file 160 output from the speech recognition apparatus 120. Referring to FIG. 5, phoneme string file 160 includes a plurality of sets of phoneme strings obtained as a result of speech recognition and time information for knowing the duration of each phoneme string. In FIG. 5, the duration is shown in milliseconds.

  The animation reproduction unit 98 generates six face images including a face image corresponding to five phonemes (/ a /, / i /, / u /, / e /, / o /) and an expressionless face image. Using a face data file storage unit 132 that stores a face data file held as a wire frame model, and a parameter for creating a face image in a key frame created by the key frame data creation unit 136, a predetermined time point constituting an animation An interpolation function storage unit 134 for storing an interpolation function used for generating a face image of a frame of the frame from six face images stored in the face data file storage unit 132, and a key frame data generation unit 136. Key frame data, a face data file stored in the face data file storage unit 132, and an interpolation function storage unit 134 Using a stored interpolation function, and a animation generation unit 138 for generating the interpolated frame of the facial image at a given point in the animation.

  An example of a face image stored in the face data file storage unit 132 is shown in FIG. 3A to 3E are face images corresponding to phonemes / a /, / i /, / u /, / e /, and / o /, respectively, and FIG. It is a corresponding face image. In this specification, these images are expressed as face images / A /, / I /, / U /, / E /, / O /, and / φ /, respectively.

  In this embodiment, the face images / A /, / I /, / U /, / E /, / O / are all based on the face image / φ /, and each feature point is the face image. In the defined three-dimensional space, it is defined by three-dimensional vector information indicating how far the corresponding feature point of the face image / φ / has moved. Accordingly, for example, an intermediate face image can be defined between the face image / A / and the face image / φ /. In the present embodiment, the concept of “blend rate” is introduced in order to define an intermediate face image between a specific face image and the face image / φ /. The blend ratio is defined as 100% for a specific face image and 0% for a face image / φ /, and the intermediate face image at the ratio of the amount of movement of feature points from the face image / φ / to the specific face image. It represents. Accordingly, when the face images / A /, / I /, / U /, / E /, / O / are assigned to phonemes as they are, the blend ratio is 100%. A face image / A / with a blend ratio of 50% is a face whose ratio of the amount of movement of feature points from the face image / φ / is 50% of the amount of movement of feature points of the face image / A /. Refers to the image. If the movement amount of the feature point of the face image is represented by a vector starting from the position at the face image / φ /, the face image with the blend rate B% is a vector representing each feature point, and the direction is the blend rate 100%. This corresponds to a face image that is equal to the vector of the face image and is reduced in length corresponding to the blend ratio B%.

  Interpolation between two face images will be described later.

  FIG. 4 shows a part of an example of the mapping table stored in the mapping table storage unit 130. Referring to FIG. 4, in the present embodiment, mapping table storage unit 130 converts phoneme / a / to face image / A /, phoneme / b / to face image / φ /, and phoneme / d / to face. The phoneme / e / is associated with the image / U / and the face image / E /, respectively. In the mapping table, a phoneme is prepared for a face image prepared in advance for a phoneme, such as face images / A /, / I /, / U /, / E /, / O / shown in FIG. Make sure to associate them. Otherwise, the obtained moving image of the face will be inconsistent with the utterance content. A phoneme that closes the lips, such as / b /, / m /, is associated with an expressionless face image / φ /. The other phonemes are appropriately assigned to the face images that are considered to have the closest mouth shape among the six face images described above.

  Referring to FIG. 2 again, the animation playback unit 98 further includes an audio file storage unit 140 for storing an audio file storing the audio data output from the audio recording unit 114, and a face sequentially given from the animation generation unit 138. An output unit 142 for synchronizing the image and the audio from the audio file stored in the audio file storage unit 140 to the monitor 102 and the speaker 100 in synchronization with each other, the text selection unit 112 of the input instruction unit 94, and the audio The recording unit 114, the key frame data generation unit 136 of the key frame data generation unit 96 and the voice recognition device 120, and the animation generation unit 138 and the output unit 142 of the animation reproduction unit 98 are operated in a predetermined sequence, and by their cooperation Animation generation And a sequence control unit 144 for controlling these to achieve a stem.

  FIG. 6 shows details of the configuration of the key frame data creation unit 136 of FIG. Referring to FIG. 6, key frame data creation unit 136 maps the face image to each phoneme in the phoneme string data from speech recognition apparatus 120 with reference to mapping table storage unit 130, and the mapped face A mapping processing unit 180 for adding and outputting an image identifier and a blend rate “100%”, and a duration length, a corresponding face image identifier and its blend rate output by the mapping processing unit 180 are attached. Durations output by the blend rate adjustment unit 182 based on the duration length and the blend rate adjustment unit 182 based on the duration length for adjusting each blend rate of the phoneme sequence based on the duration length of each phoneme The blending rate of the phoneme string with the blending rate adjusted according to the length, the corresponding face image, and the duration time, and the power level in the duration of each phoneme For adjusting the basis, and a blend ratio adjustment unit 184 by the power. The output of the blend ratio adjustment unit 184 by power is a phoneme string in which each phoneme is assigned with its duration, the corresponding face image, and the blend ratio adjusted by the duration and power. This phoneme string is key frame data. In the present embodiment, a key frame refers to a frame when the frame is created at the start time of the duration of each phoneme.

  FIG. 7 shows a more detailed block diagram of the animation generation unit 138. Referring to FIG. 7, the animation generation unit 138 includes a face image to which a predetermined blend ratio is assigned in two key frames, a time corresponding to the two key frames, and the interval between the two key frames. Then, given the time at which the animation should be generated (for convenience, it is called “generation time”. The generation time is expressed as a relative time with reference to the time of two key frames). An interpolation processing unit 204 for generating a face image at the generation time from the face images of the two key frames by interpolation using an interpolation function stored in the interpolation function storage unit 134 and outputting the generated face image to the output unit 142. When the predetermined generation time is determined, two key frames sandwiching the generation time are determined, and the face image data and blend ratio in those key frames are determined. When the interpolation processing unit 204 is provided with information indicating the relative position of the generation time between the times of the two key frames to generate a face image at the generation time, and the generation of the face image by the interpolation processing unit 204 is finished. The animation generation control unit 200 has a function of repeating the process of creating the next face image with the current time as the next generation time, and the timer 202 that the animation generation control unit 200 refers to in order to determine the time. This interpolation processing and animation generation processing will be described later.

  With reference to FIGS. 8 to 14, the face animation creation process by the key frame data creation unit 136 and the animation generation unit 138 according to the present embodiment will be described in more detail.

  For example, assume that a phoneme string file 160 as shown in FIG. 5 is given. In this case, the output of the mapping processing unit 180 shown in FIG. 6 is shown in FIG. Referring to FIG. 8, on the time axis, the utterance period of each phoneme / a /, / i /, / u /, / e /, / o / has durations 300, 300, 100, 30 and 250 ( All are assigned in milliseconds). The start time of each period is a key frame. The blend rate at each key frame is 100%.

FIG. 9 shows an example of a result in the middle of processing the phoneme string as shown in FIG. 8 by the blend rate adjusting unit 182 based on the duration time. Referring to FIG. 9, first, among each phoneme, a phoneme having a duration time smaller than a predetermined threshold is integrated into the immediately preceding phoneme period. In the example shown in FIG. 8, if the phoneme / e / duration is 30 milliseconds and the threshold is 50 milliseconds, the phoneme / e / is deleted and its duration is immediately before it. Phoneme / u /. Accordingly, as shown in FIG. 9, the phoneme string is / a /, / i /, / u /, / o /, and the durations thereof are 300, 300, 130, and 250 (milliseconds), respectively. Since one phoneme is deleted, the number of key frames is reduced from five to four. In the following description, the times corresponding to these key frames are T ₀ , T ₁ , T _2, and T ₃ , respectively, and the time of the key frame immediately after the last phoneme / o / is T ₄ . Hereinafter, a key frame defined by the start time T _k of the k-th phoneme is generally referred to as “key frame T _k ”.

The blend rate adjustment unit 182 according to the duration time further adjusts the blend rate assigned to each key frame according to the duration time. Specifically, the blending rate adjustment unit 182 based on the duration length finds the maximum duration length L _MAX in the phoneme string, and adjusts the blending rate of the duration length of each phoneme by the following equation (1). To do.

ただし、Ｂ（ｎ）はｎ番目の音素のブレンド率を、Ｌ（ｎ）はｎ番目の音素の継続時間長を、それぞれ表す。Ｃ_１は所定の定数で、例えば短い継続時間長の音素を削除したときに使用されるしきい値と同程度の大きさに選ばれる。こうしてブレンド率が継続時間長により調整された音素列の、時間軸上での配置とそのブレンド率とを図１０に模式的に示す。図１０において、例えば、調整後の音素／ａ／，／ｉ／，／ｕ／，／ｏ／のブレンド率はそれぞれａ、ａ、ｂ、及びｃ（％）である。

Here, B (n) represents the blend rate of the nth phoneme, and L (n) represents the duration of the nth phoneme. C ₁ is a predetermined constant, and is selected to be approximately the same as the threshold used when a phoneme having a short duration is deleted, for example. FIG. 10 schematically shows the arrangement on the time axis and the blend rate of the phoneme string in which the blend rate is adjusted by the duration time. In FIG. 10, for example, the blend ratios of the adjusted phonemes / a /, / i /, / u /, and / o / are a, a, b, and c (%), respectively.

  Similarly to the blend rate adjusting unit 182 based on the duration time, the blend rate adjusting unit 184 by power adjusts the blend rate of each phoneme by the sound power in the duration of each phoneme. Specifically, first, a phoneme having power less than or equal to a predetermined threshold is deleted, and the duration is integrated with the duration of the previous phoneme. The head of each continuous period thus obtained is a key frame. Each key frame is assigned a blend rate. The blend ratio adjustment unit 184 by power adjusts this blend ratio (the blend ratio of the nth phoneme is set to B (n) as before) by the following formula (2).

ただしＰ_ＭＡＸは全体でのパワーの最大値であり、Ｐ（ｎ）はｎ番目の音素の継続期間のパワーであり、Ｃ_２は所定のしきい値である。このしきい値も、前述した音素の削除のときに使用されたしきい値と同程度の大きさに選ばれる。

However, P _MAX is the maximum value of the overall power, P (n) is the power of the duration of the nth phoneme, and C ₂ is a predetermined threshold value. This threshold value is also selected to be as large as the threshold value used when deleting the phonemes described above.

FIG. 11 schematically shows the phoneme string finally obtained in this way, its duration, and the blend ratio after adjustment in each key frame. Referring to FIG. 11, the phonemes of key frames T ₀ , T ₁ , T ₂ , and T ₃ are / a /, / i /, / u /, and / o /, respectively, and the blend rates are a ′ and a ′, respectively. ”, B ′ and c ′ (where a ′ ≦ a, a” ≦ a, b ′ ≦ b and c ′ ≦ c), and the durations are 300, 300, 130 and 250 (milliseconds), respectively. .

  In this way, key frame data is created.

  Next, the interpolation processing by the interpolation processing unit 204 shown in FIG. 7 will be described. Various interpolation functions are conceivable as the interpolation function stored in the interpolation function storage unit 134. In the present embodiment, the function that gives linear interpolation is used with emphasis on the ease of calculation processing and high speed. The concept of linear interpolation will be described with reference to FIG.

Referring to FIG. 12, the horizontal axis represents the time axis, and the blend rate at time T ₀ , T ₁ , T ₂ , T ₃ . As shown by line segments 220, 222, 224, and 226, the interpolation function in the present embodiment is a point of the blend rate at each key frame and the blend rate “0” at the time of the adjacent key frame. Is a function that linearly interpolates the blend rate at each time along the line segment connecting. That is, this function performs linear interpolation so that the rate on one side is 100% and the rate on the other side is 0%.

For example, it is assumed that an intermediate time t between time T ₀ and time T ₁ is the generation time. The phonemes at key frames T ₀ and T ₁ are / a / and / i /, respectively. The blend rate at each key frame is calculated by a blend rate adjustment unit 184 based on power. The blend ratio a 'at time T _0, the blend ratio point and a line segment 220 connecting the "0" at time T _1, the blend ratio of keyframes T ₀ alpha is linearly interpolated at time t. Similarly, the blend rate β of the key frame T ₁ at time t along the line 222 connecting the point of the blend rate “0” at time T ₀ and the point of the blend rate a ″ at time T _1. Is calculated.

The blend rate of the key frame T _{0 at} the time T ₀ is B (T ₀ ), the blend rate of the key frame T _{1 at} the time T ₁ is B (T ₁ ), and the key frame T _{0 at} the time t, obtained by interpolation, T ₁ of the blend ratio, respectively, and α and beta. Then, α and β are obtained by the following equation (3).

本実施の形態では、このようにして算出された二つのブレンド率（例えばα及びβ）を用い、図１３に示されるようにして時刻ｔにおける顔画像を作成する。

In the present embodiment, using the two blend rates (for example, α and β) calculated in this way, a face image at time t is created as shown in FIG.

Now, X (T ₀ ) is a three-dimensional vector whose element is a movement amount of each feature point of the face image at the key frame T ₀ on the basis of the corresponding feature point at the face image / φ /, and similarly the key frame. the three-dimensional vector in by T ₁ and X _{(T 1).} Then, the three-dimensional vector X (t) whose element is the movement amount of each feature point of the face image at T ₀ ≦ t ≦ T _{1 with} respect to the corresponding feature point of the face image / φ / is expressed by the following formula ( It is calculated by the vector weighted sum represented by 4).

補間処理部２０４は、こうした計算を顔画像の各特徴点に対して実行する。後述するようにこうした演算はグラフィックプロセッサユニット（ＧＰＵ）が得意とするところである。従ってアニメーション生成システム８０は、ＧＰＵを備えていることが望ましい。

The interpolation processing unit 204 performs such calculation for each feature point of the face image. As will be described later, such a calculation is a specialty of the graphic processor unit (GPU). Therefore, it is desirable that the animation generation system 80 includes a GPU.

The animation generation control by the animation generation control unit 200 will be described. Referring to FIG. 14, it is assumed that animation generation control unit 200 starts creating an animation from time t equal to time T ₀ of the first key frame. That is, at time t ₀ (= T ₀ ), the animation generation control unit 200 gives a face image generation instruction 240 to the interpolation processing unit 204. In other words, it is the generation time _t = t _0.

In this case, first, an integer k that satisfies T _k-1 ≦ t ≦ T _k is searched. Here, since t = t ₀ = T ₀ , k = 1. The interpolation processing unit 204 sets the blend rate a ′ (see FIG. 11) at this time to the three-dimensional vector X (T ₀ ) of each feature point constituting the face image / A / of the phoneme / a / at the time T ₀ . Multiply. Further, the three-dimensional vector X (T ₁ ) of each feature point constituting the face image / I / of the phoneme / i / at time T ₁ is multiplied by the blend rate a ″ (see FIG. 11) at this time. Next, the processing unit 204 calculates α and β using Equation (3), where t = T _0, so that α = B (T ₀ ) and β = 0. The face image 242 at time t is generated and output according to 4).

It is assumed that this generation process takes time s ₁ . When the face image 242 is generated and output, the interpolation processing unit 204 gives a completion notification 244 indicating that the processing is completed to the animation generation control unit 200. This time t ₂ as a new generation time t.

Animation generation control unit 200, in response to receiving the completion notification 244 in the new generation time t = t _2, the two keyframes sandwiching the generation time t, the time T ₀ and T ₁ in the example shown in FIG. 14 , And the face images / A / and / I / in these two key frames, the two times T ₀ and T _1, and the relative time t = t ₂ in the times T _{0 to} T ₁ Time is given to the interpolation processing unit 204, and a face image generation instruction 246 is given.

In response to this instruction, the interpolation processing unit 204 generates and outputs a face image 248 at the generation time t = t ₂ over time s ₂ . At this time, the interpolation processing unit 204 gives a completion notification 250 to the animation generation control unit 200. The time t ₃ at this time is a new generation time t.

Then, the animation generation control unit 200 performs a process similar to that performed at the previous generation time t ₂ for the new generation time t = t ₃ , and sends a face image generation instruction 252 to the interpolation processing unit 204. Give to. Similarly, the output face image 254 after a time s _3, the completion notification 256 is given to the animation generation control unit 200 at time t _3, the face image generated at time t ₃ from the animation generation control unit 200 in response to the following Is given to the interpolation processing unit 204. The same applies hereinafter.

  In other words, in the present embodiment, the animation generation control unit 200 controls the timing for animation generation so that the interpolation processing unit 204 always exhibits its full potential.

<Operation>
2 to 14, the animation generation system 80 described above operates as follows. The following operations of the respective units are performed in a predetermined sequence by the control by the sequence control unit 144 shown in FIG. 2, but for the sake of easy understanding, the control of the sequence control unit 144 will not be described below.

  The face image of the animation character is prepared in advance for the above six types of phonemes and stored in the face data file storage unit 132. A mapping table for mapping the face image to each phoneme is also prepared in advance and stored in the mapping table storage unit 130. A program for realizing the interpolation function is also prepared in advance and stored in the interpolation function storage unit 134. Further, several types of transcription texts for user utterances are prepared in advance and stored in the text storage unit 110.

  The text selection unit 112 reads all the transcription text stored in the text storage unit 110, displays the text on the text selection interface 90, and displays a message prompting to select one of them.

  When the user selects any text, the text selection unit 112 provides the text to the key frame data creation unit 136 and displays a message on the text selection interface 90 prompting the user to speak the text. At the same time, the text selection unit 112 activates the voice recording unit 114 and starts recording a voice signal from the microphone 92.

  The audio recording unit 114 creates audio data obtained by framing the input audio with a predetermined frame length and a predetermined shift length, and stores the audio data as an audio data file in the hard disk.

  When the recording of the audio signal is completed, the text selection unit 112 and the audio recording unit 114 give the written text and the audio data file to the key frame data generation unit 136 and the audio recognition device 120, respectively.

  The key frame data creation unit 136 and the speech recognition device 120 operate on this data as follows. First, the speech recognition device 120 performs phoneme segmentation on the speech data file given from the speech recording unit 114 with reference to the transcription data, and specifies the phoneme sequence file 160 (the duration length is specified) as shown in FIG. (Including possible time information). The speech recognition apparatus 120 gives the data of the phoneme string file 160 to the mapping processing unit 180 of the key frame data creation unit 136.

  Referring to FIG. 6, mapping processing unit 180 of key frame data creation unit 136 assigns a face image identifier to each phoneme in phoneme sequence file 160 with reference to mapping table storage unit 130, This is given to the blend rate adjustment unit 182 according to the duration time.

  The blend rate adjustment unit 182 based on duration length compares the duration length of each phoneme with a threshold value, deletes a phoneme having a duration length less than the threshold value, and continues the duration of the previous phoneme. Integrate into the period. The blend rate adjustment unit 182 according to the duration time further adjusts the blend rate of each phoneme based on the maximum value of the phoneme duration and the duration of the phoneme according to the equation (1). The blend rate adjustment unit 182 based on the duration length gives the phoneme string with the duration time, the face image identifier, and the blend rate created in this way to the blend rate adjustment unit 184 based on power.

  The power blend ratio adjusting unit 184 has a power value during a period of less than a predetermined threshold among the phonemes of the phoneme string given from the blend ratio adjusting unit 182 by the duration time, Delete the phoneme. Then, the duration of the phoneme is integrated with the duration of the previous phoneme.

  The power blend ratio adjusting unit 184 further adjusts the blend ratio of each phoneme according to the formula (2) based on the maximum power value and the power of each phoneme. The power-based blend rate adjustment unit 184 supplies key frame data composed of phoneme strings with the blend rate adjusted in this way to the animation generation control unit 200 shown in FIG.

Referring to FIG. 7, animation generation control unit 200 first sets generation time t to the time of the first key frame (start time of the first phoneme) in the given key frame data, and T _k−1 ≦ Search for an integer k such that t <T _k . In this case, since t = T ₀ = T _k−1 , k = 1. The animation generation control unit 200 reads the face image data corresponding to the key frames T ₀ and T ₁ from the face data file storage unit 132, and generates the times T ₀ and T ₁ , the generation time t, and the key frames T ₀ and T _1. Is provided to the interpolation processing unit 204.

Based on the given times T ₀ , T ₁ , the blend rate of the key frames T ₀ , T ₁ , and the value of the generation time t, the interpolation processing unit 204 determines the blend rate of the key frames T ₀ and T ₁ . The blending ratios α and β at the generation time t are calculated using the interpolation function (formula (3)) stored in the interpolation function storage unit 134, respectively. The interpolation processing unit 204 further uses the calculated blend rates α and β and the feature point vectors X (T ₀ ) and X (T ₁ ) of the face image data in the key frames T ₀ and T ₁ as described above. Using the equation (4), each feature point vector x (t) of the face image at the generation time t is calculated and given to the output unit 142. The output unit 142 displays this image on the display 102. The output unit 142 starts reproduction of the audio file stored in the audio file storage unit 140 in synchronization with the display of the face image.

When the calculation of the face image at the generation time t is completed, the animation generation unit 138 gives a signal indicating the end of the process to the animation generation control unit 200. When the animation generation control unit 200 receives this signal, it determines the time at that time with reference to the timer 202. The animation generation control unit 200 sets this time as a new generation time t, and determines an integer k that satisfies T _{k −1} ≦ t <T _k . Then, the data of the times T _k−1 , T _k , the generation time t, and the key frames T _k−1 and T _k are given to the interpolation processing unit 204, and the generation of the face image at the time t is executed.

In the same manner as in the previous cycle, the interpolation processing unit 204 converts the given times T _k−1 , T _k , the blend rate of those key frames T _k−1 , T _k , and the value of the generation time t. Based on the blending ratios α and β at the generation time t with respect to the blending ratios of the key frames T _k−1 and T _k , based on the interpolation function (Equation (3)) stored in the interpolation function storage unit 134, respectively. The interpolation processing unit 204 further calculates the calculated blend ratios α and β and the feature point vectors X (T _k−1 ) and X (T _k ) of the face image data in the key frames T _k−1 and T _k . Then, each feature point vector x (t) of the face image at the generation time t is calculated using Expression (4), and is given to the output unit 142. The output unit 142 displays this image on the display 102. The reproduction of the audio file storage unit 140 is continued in synchronization with the image output.

  When the calculation of the face image at the generation time t is completed, the interpolation processing unit 204 gives a signal indicating the calculation to the animation generation control unit 200. When receiving this signal, the animation generation control unit 200 obtains the time at that time with reference to the timer 202. Then, the time is set as a new generation time t.

  Thereafter, the same processing is repeated. When the generation time t exceeds the audio recording time, the animation generation system 80 ends the animation generation process, and the subsequent state returns to the state at the time of the first writing and display at the time of text selection.

  When the user selects and reads out a certain utterance text in this way, the shape of the mouth is deformed in synchronization with the voice data using the face image data stored in the face data file storage unit 132 based on the voice data. The animation of the facial image to be obtained is obtained. The first recorded sound is also played back in synchronization with the face image, so the animated character appears to be speaking. As a result, it is possible to obtain an animation in which the character speaks with the voice of the user.

  In the present embodiment, only face images corresponding to limited phonemes are prepared, but a sufficiently natural animation can be obtained by mapping an appropriate face image to each phoneme using a mapping table. Can be obtained. When the phoneme duration is extremely short or the power is extremely small, image generation is omitted for the phoneme. Usually, the actual movement of the face when generating such phonemes is very small. Therefore, this omission has an effect that the animation of the obtained face image can be felt close to natural movement. Furthermore, using the concept of blend ratio, the amount of facial deformation (each feature point is moved from each feature point position of the reference image (image without expression)) according to the strength of each phoneme utterance Amount). Therefore, it is possible to obtain an animation of a face image with natural movement according to the strength of phoneme utterance. A face image between adjacent key frames is obtained by interpolating the face images of adjacent key frames with a weighted sum corresponding to the blend rate in the key frames, the time of the key frames, and the generation time of the images. ing. Accordingly, the mouth shape changes smoothly from the phoneme to the phoneme, and the animation of the obtained face image feels natural.

[Second Embodiment]
If there is a computer with sufficient performance, the animation generation system 80 according to the first embodiment described above can be realized with only one computer. However, in order to complete the work in a relatively short time, it is practical to use a plurality of computers.

  FIG. 15 shows a schematic configuration of an animation generation system 280 according to the second embodiment of the present invention. Referring to FIG. 15, this animation generation system 280 receives a voice input from an unspecified user, and performs a voice input computer 292 that performs a process of starting generation of an animation in animation generation system 280, and a voice input A voice recognition server 294 for generating key frame data by performing phoneme segmentation on the voice input by the computer 292, and key frame data output by the voice recognition server 294 in response to voice input by the voice input computer 292 And an animation display computer 296 for creating and displaying an animation of a face image of a predetermined character whose mouth shape changes in synchronization with the input voice. The voice input computer 292, the voice recognition server 294, and the animation display computer 296 are all communicable with each other via a network 290 using a predetermined protocol.

  Compared with FIG. 2, the computer 292 for voice input is the input instruction unit 94 in FIG. 2, the voice recognition server 294 is in the voice recognition device 120 in FIG. 2, and the animation display computer 296 is in the animation playback unit 98 in FIG. , Respectively. The functional configurations of the voice input computer 292, the voice recognition server 294, and the animation display computer 296 are the same as the configurations of the input instruction unit 94, the voice recognition device 120, and the animation playback unit 98 in FIG. The details are not repeated here.

  The computer 292 for voice input has a touch panel 300 and a microphone 302. The animation display computer 296 includes a monitor 310 having a speaker 312 and a computer housing 314 disposed under the monitor 310.

  FIG. 16 shows a hardware configuration of the animation display computer 296. Referring to FIG. 16, in addition to the monitor 310 and speaker 312 shown in FIG. 15, an animation display computer 296 includes a CPU (Central Processing Unit) 350 and a read-only memory, both of which are arranged in the computer main body 314. (ROM) 352, read / writeable memory (RAM) 354, hard disk drive 356, DVD drive 358 in which a DVD (Digital Versatile Disk) 330 can be mounted, and GPU 360 for executing facial image calculation processing Including. These are all connected to the CPU 350 by a bus 362.

  Each of the animation display computers 296 is further arranged in the computer main body 314 and connected to the bus 362. A memory port that can be equipped with a portable memory 332 composed of a network interface (I / F) 368 and a flash memory. 366 and a sound board 364 to which a speaker 312 is connected.

  Note that the animation display computer 296 does not include a keyboard because it is not necessary to use a keyboard in the animation creation process. Of course, when the animation display computer 296 is used as a normal computer, an input device such as a keyboard and a mouse can be connected to the computer main body 314.

  Although not shown in the drawing, the hardware configuration of the voice input computer 292 and the voice recognition server 294 is substantially the same as that of the animation display computer 296. Speaking of differences, in the computer 292 for voice input, the monitor 310 and the input device are integrated to form the touch panel 300, the computer 292 for voice input further includes the microphone 302, and the voice. For example, the GPU 360 is not required in the input computer 292 and the speech recognition server 294.

  FIG. 17 is a flowchart showing a control structure of a computer program for causing the voice input computer 292 to operate as the input instruction unit 94 shown in FIG. 2 by being executed by the voice input computer 292.

  Referring to FIG. 17, when the computer 292 for voice input is turned on and this program is started, an initialization process is executed in step 400. In this processing, resources necessary for the processing are secured and initialized in the computer 292 for voice input, the communication function is confirmed, and the utterance text is read from the utterance text file.

  When the initialization process is finished, in step 402, the animation display computer 296 is notified that the preparation of the computer 292 for voice input is completed. Subsequently, at step 404, whether or not a preparation completion notification indicating that the voice input computer 292, the voice recognition server 294, and the animation display computer 296 are all ready is received from the animation display computer 296. Determine. When the preparation completion notification is received, the process proceeds to step 406. The determination process in step 404 is repeated until a preparation completion notification is received.

  In this way, waiting for the preparation completion notification from the animation display computer 296 is not completed in all cases, assuming that the voice input computer 292, the voice recognition server 294, and the animation display computer 296 are activated at the same time. This is because the animation generation system 280 cannot function as a whole.

  Subsequently, in step 406, several utterance texts are displayed on the touch panel 300, and an input waiting message “Please select one text” is displayed. Then, in step 408, the state waits for input. If there is an input, i.e. the text is selected by the user, the process proceeds to step 410.

  In step 410, a message prompting the user to speak the selected text is displayed and recording begins. When recording ends, the process proceeds to step 412.

  In step 412, the recording completion is notified to the animation display computer 296. In the following step 414, it is determined whether or not a processing start notification has been received from the animation display computer 296. The process start notification is a message indicating that execution of the phoneme segmentation process in the speech recognition server 294 and the animation generation process in the animation display computer 296 is started.

  In step 416, a display indicating that processing is in progress is displayed on touch panel 300. In step 418, the voice data recorded in step 410 and the corresponding text data (transcription data) are transmitted to the animation display computer 296. In step 420, the process waits until a processing completion notification is received from the animation display computer 296. The process completion notification is a notification indicating that the phoneme segmentation process and the subsequent key frame data creation process have been completed for the audio data transmitted to the animation display computer 296 in step 418.

  When the processing completion notification is received, an animation output command is transmitted to the animation display computer 296 in step 422. As will be described later, the animation display computer 296 starts animation generation processing and output processing in response to this output command. In step 424, the process waits until an output notification is received from the animation display computer 296. If an output display is received, the process proceeds to step 426. In step 426, a message indicating that the animation is being output on the monitor 310 of the animation display computer 296 is displayed on the touch panel 300. In step 428, the output computer 296 waits for an output completion notification indicating that the animation output processing has been completed. When the output completion notification is received, the generation and display of the animation for the sound recorded in step 410 are all completed. Accordingly, control returns to step 406 to wait for the next user input.

  The computer 292 for voice input repeats the above processing.

  FIG. 18 is a flowchart of processing executed by the voice recognition server 294. When this program is started, initialization processing is executed in step 440. When the initialization process is completed, in step 442, a notification indicating that the preparation of the voice recognition server 294 is completed is transmitted to the animation display computer 296.

  In step 444, it is determined whether or not a phoneme segmentation request has been received from the animation display computer 296. Phoneme segmentation is processing similar to speech recognition processing, and refers to processing that divides input speech into phonemes using an acoustic model. When the request is received, the process proceeds to step 448.

  In step 448, the voice display server 294 is notified to the animation display computer 296 that the phoneme segmentation process has started.

  In the subsequent step 450, in accordance with the request, voice data and transcription data to be subjected to phoneme segmentation are acquired from the animation display computer 296. When the acquisition is completed, a notification indicating that the reception of the target data is completed is transmitted to the animation display computer 296 in step 452. In step 454, phoneme segmentation processing using an acoustic model (not shown) and the received transcription data is executed on the received voice data. In this process, since transcription data exists, it is possible to perform accurate phoneme segmentation.

  When the phoneme segmentation is completed and the generation of the phoneme string file is completed, in step 456, the generation of the phoneme string file is notified to the animation display computer 296.

  Furthermore, based on this phoneme string file, in step 458, key frame data generation processing is executed. Details of the key frame data generation process will be described later with reference to FIG. When the key frame data generation process is completed, the animation display computer 296 is notified in step 460 that the key frame data generation has been completed. In step 462, the phoneme string file and the key frame data are transmitted to the animation display computer 296. In step 464, the animation display computer 296 is notified that all processing in the voice recognition server 294 has been completed, and the process returns to step 444.

  FIG. 19 is a flowchart of the key frame data creation process executed in step 458 of FIG. Referring to FIG. 19, in step 480, it is determined whether or not there is a phoneme having a duration less than a predetermined threshold or a phoneme having a power less than a predetermined threshold in a given phoneme string. judge. If there is, in step 482, the phoneme is deleted, the duration of the phoneme is integrated with the duration of the previous phoneme, and the process returns to step 480. When there is no phoneme as described above, the process proceeds to step 484.

  In step 484, 100% is set as the initial value of the blend ratio for all phonemes constituting the given phoneme string. In the following step 486, any one of the face images / A / ˜ / φ / shown in FIG. 3 is used for each phoneme in the phoneme string using the mapping table stored in the mapping table storage unit 130 shown in FIG. And the identifier of the face image is attached to the phoneme. The face image assigned in this way becomes a key frame whose frame time is the start time of the phoneme.

Subsequently, in step 488, all the given phoneme strings are examined, and the maximum duration time and the maximum power of the phonemes are searched. The searched maximum duration length is L _MAX , and the maximum power is P _MAX .

  In step 490, the blend ratio of each phoneme is updated by the above-described equation (1). In Equation (1), B (n) represents the blend rate of the nth phoneme. Similarly, in step 492, the blend ratio of each phoneme is updated by the above-described equation (2).

  In step 494, the phoneme sequence with the blend ratio calculated as described above, the identifier of the corresponding face image, and the time information is output to a file as key frame data, and key frame data creation processing is performed. finish.

  FIG. 20 is a flowchart showing a control structure of a computer program that realizes animation generation control processing executed by the animation display computer 296. Referring to FIG. 20, when the animation generation control process is activated, an initialization process is performed in step 500, and a preparation completion notification from voice input computer 292 and voice recognition server 294 is waited in step 502.

  When the preparation completion notification is received from the voice input computer 292 and the voice recognition server 294, in step 504, a preparation completion notification indicating that the entire animation generation system 280 is ready is transmitted to the voice input computer 292. To do. In step 506, the process waits until a recording completion notification is received from the computer 292 for voice input.

  When the recording completion notification is received, in step 508, a process start notification indicating that the voice recognition server 294 and the animation display computer 296 start executing a series of processes for creating an animation to the computer 292 for voice input. Send. Subsequently, in step 510, the process waits until the text data and voice data are received from the voice input computer 292. When these data are received, the process proceeds to step 512.

  In step 512, the transcription text data and the speech data received in step 510 are transmitted to the speech recognition server 294, and a phoneme segmentation is requested. In step 514, the process waits until key frame data obtained as a result of phoneme segmentation is received from the speech recognition server 294. When the key frame data is received, the processing for animation generation in step 520 and subsequent steps is executed.

In step 520, in this embodiment, as the time (generation time) t of the first frame of the animation of the facial image, selects the time T ₀ of the first phoneme of the phoneme sequence.

Subsequently, in step 522, _k that satisfies T _{k −1} ≦ t <T _k is determined with respect to the frame generation time t determined in the immediately preceding step. However, T _k indicates the start time of the period of the k-th phoneme in the phoneme string. For example, if t = T ₀ , T ₀ ≦ t <T ₁ , so k = 1.

In Step 524 is followed, the time _{T k} and _{T k-1,} a key frame _{T k-1,} and the blend ratio of _{T k,} and generation time t, the face corresponding to the phoneme at the time _{T k} and _{T k-1} The image data is transferred to the GPU 360, and a request is made to generate a face image at the generation time t by interpolation.

In response, the program GPU360 is executed, the generation time t, the blend ratio is the interpolation operation from a blend ratio keyframe T _k-1 alpha, and a keyframe T blend ratio that is the interpolation operation from a blend ratio of _k β is calculated by the above-described interpolation formula (3), and each feature point vector X (t) of the face image at the generation time t is converted to each feature point vector X of the face image at the key frames T _k−1 and T _k . Using (T _k-1 ) and X (T _k ), and α and β, the calculation is performed by the vector weighted sum according to the above-described equation (4). When this calculation is completed for all feature points of the face image, the GPU 360 outputs the generated face image at time t, and further transmits a process end notification to the CPU 350.

  Referring to FIG. 20, the animation generation control processing program is in a waiting state until an end notification is received from GPU 360 in step 526. When the end notification is received, the process proceeds to step 528.

  In step 528, the time of the timer 202 is read. This time is set as a new generation time t. Subsequently, in step 530, it is determined whether or not the generation time t is later than the final recording time. If the generation time t is after the recording time, the process is terminated. Otherwise, the process returns to step 522 to generate face image data at this new generation time t.

  Thereafter, the processing from step 522 to step 530 is repeated until the generation time t becomes larger than the recording time. When the generation time t becomes larger than the recording time, the process proceeds to step 532.

  In step 532, a notification indicating that the processing in the voice recognition server 294 and the animation display computer 296 is completed is transmitted to the voice input computer 292. The voice input computer 292 receives this notification at step 428 in FIG. 17, and in response to this, returns to step 406, and the above-described series of processing is repeated from the voice input.

  The program having the control structure shown in the flowchart in FIG. 17 is the computer 292 for voice input, the program having the control structure shown in the flowchart in FIGS. 18 and 19 is in the voice recognition server 294, and the control structure shown in the flowchart in FIG. The animation generation system 280 having the same functions as those of the animation generation system 80 according to the first embodiment can be realized by executing each of the stored programs on the animation display computer 296.

  Note that, when the animation generation system 80 according to the first embodiment is realized by a computer, a program similar to the computer program having the control structure shown in FIGS. 17 to 20 can be used.

<Operation>
The operation of the animation generation system 280 according to the second embodiment is the same as that of the animation generation system 80 according to the first embodiment. Therefore, details thereof will not be repeated here.

  In this embodiment, processing is distributed to each computer. Therefore, the performance of each computer may not be so high. A high-performance voice recognition server 294 is prepared, and a key frame data creation request from a plurality of voice input computers 292 and animation display computers 296 is sent to a single voice recognition server 294. Can also be processed.

  Further, in this embodiment, the computer 292 for voice input and the computer 296 for animation display are separate computers, but they may be realized by a single computer.

  It is a design matter at the time of animation production which face image corresponding to what phoneme is prepared and how many face images are prepared. Also, what face image is mapped to which phoneme is a design item at the time of animation production. In addition, the phoneme set varies depending on the target language, and therefore the mapping is naturally different.

  In the above-described embodiment, the phoneme and the face image are mapped so that one face image always corresponds to a certain phoneme, but this need not be the case. That is, even with the same phoneme, different face images may be assigned depending on the phonemes before and after.

  In the above-described embodiment, the equations (1) and (2) are used for calculating the blend ratio. However, the present invention is not limited to those using the formulas (1) and (2). As long as the duration or power becomes shorter, the blend rate becomes lower, that is, as long as it is a monotonic function with respect to the duration and power, any function can be used to calculate the blend rate. Good. In addition, the blend rate may be determined in consideration of not only the duration time and power but also other audio feature amounts.

  In the above-described embodiment, an interpolation function corresponding to a linear equation as shown in FIG. 12 is used. However, the present invention is not limited to such an embodiment. As an interpolation function, a second-order or higher order polynomial with respect to time may be used, or a nonlinear function may be used. In the present embodiment, any interpolation function may be used as long as the blend rate is the highest at the time corresponding to the key frame, and the blend rate is lowered as the distance from the time is increased. A plurality of interpolation functions may be prepared so that the user can use them by switching.

  In the above embodiment, the position of the key frame is the head position of the duration of each phoneme, but the present invention is not limited to such an embodiment. The position of the key frame may be in the middle of the duration of each phoneme. The position of the key frame may be arbitrarily changed by the user.

  In the above-described embodiment, if the duration or power of a phoneme in the phoneme string is smaller than the threshold, the phoneme is deleted, and the duration is changed to the duration of the previous phoneme. Integrated. By doing so, it is possible to obtain an effect that the change in the mouth shape is smooth and natural.

  However, the present invention is not limited to such an embodiment. For example, only the duration of a phoneme may be considered, or only power may be considered. Alternatively, the phoneme may be deleted when both the duration time and power are smaller than the threshold values. Switching between them may be performed.

  Furthermore, in the above-described embodiment, the voice that is finally reproduced together with the animation is the voice of the user recorded first. However, the present invention is not limited to such an embodiment. Since the mouth shape is mainly determined in relation to the phoneme, some processing may be added to the user's voice as long as no significant change is made to the position of the phoneme. Even in this case, the characteristics of the user's utterance are often utilized for the finally reproduced audio, and more various animations can be created.

  In the above-described embodiment, after recording the transcribed text utterance by the user, a key data file is generated. After the key data file is generated, the next face image is generated at the end of the face image creation process by the GPU 360. Like to start. Therefore, the generation of the face image is not performed in a constant cycle. By doing so, the GPU 360 can fully exhibit its performance. However, the present invention is not limited to creating a face image in this way.

  For example, as shown in FIG. 12, after obtaining a blend rate distribution by interpolation of each key frame, a series of times at which a face image should be generated at a certain frame interval is obtained, and a face image at each time is generated. Then, after all the face images are generated, they may be reproduced as an animation. In this case, if the frame interval is shortened, the processing takes a long time. Conversely, if the frame interval is increased, the motion of the animation may become awkward.

  In the above-described embodiment, an animation of a face image is created and reproduced in accordance with the user's voice. Since the transcription text of the voice is determined, phoneme segmentation can be performed with high accuracy even if the user is unspecified, and a smooth animation can be created.

  In the above embodiment, when a voice is input, an animation created based on the voice is played only once and the next voice is input. However, the present invention is not limited to such an embodiment. After inputting the sound and creating the key frame data, the animation can be reproduced any number of times based on the key frame data. In particular, in this reproduction, various animations can be generated from the same sound by changing a face image to be used, changing an interpolation function, or changing a threshold value when thinning phonemes. Therefore, it can be used as a tool for creating an animation by a so-called Presco method.

  Furthermore, the above-described embodiment generates an animation of a face image based on voice. However, the present invention is not limited to such an embodiment. Any shape can be applied as long as the shape changes with sound and mapping between the shape and a phoneme is possible. For example, it is conceivable to create a vocal tract-shaped animation in accordance with the voice or to create an animation of the articulation mechanism.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

It is a figure which shows the outline of the animation preparation process 30 by the animation preparation apparatus concerning the 1st Embodiment of this invention. It is a block diagram which shows the schematic functional structure of the animation production | generation system 80 which concerns on 1st Embodiment. 6 is a diagram illustrating an example of a face image stored in a face data file storage unit 132. FIG. It is a figure which shows a part of example of the mapping table memorize | stored in the mapping table memory | storage part 130. FIG. It is a figure which shows the structural example of the phoneme sequence file 160 which the speech recognition apparatus 120 outputs. FIG. 3 is a block diagram illustrating details of a configuration of a key frame data creation unit 136 in FIG. 2. It is a more detailed block diagram of the animation generation unit 138. It is a figure which shows the relationship between a phoneme string and a blend rate. It is a figure which shows the relationship between a phoneme string and a blend rate. It is a figure which shows the relationship between a phoneme string and a blend rate. It is a figure which shows the relationship between a phoneme string and a blend rate. It is a figure which shows the outline of the interpolation of a blend rate. It is a figure for demonstrating the vector weighted sum of the face image in a key frame. It is a figure for demonstrating the production | generation control processing of the animation by the animation production | generation control part 200. FIG. It is a block diagram which shows schematic structure of the animation production | generation system 280 concerning the 2nd Embodiment of this invention. FIG. 22 is a block diagram showing a hardware configuration of an animation display computer 296. 3 is a flowchart showing a control structure of a computer program for causing a voice input computer 292 to operate as the input instruction unit 94 shown in FIG. 2. It is a flowchart which shows the control structure of the computer program which the speech recognition server 294 performs. It is a flowchart which shows the control structure of the computer program which implement | achieves the preparation process of key frame data. It is a flowchart which shows the control structure of the computer program which implement | achieves animation production | generation control processing.

Explanation of symbols

  40 speakers, 42 audio signals, 44 scripts, 60-68 face images, 80, 280 animation generation system, 90 text selection interface, 92 microphone, 94 input instruction unit, 96 key frame data creation unit, 98 animation playback unit, 100 Speaker, 102, 310 Monitor, 110 Text storage unit, 112 Text selection unit, 114 Voice recording unit, 120 Speech recognition device, 130 Mapping table storage unit, 132 Face data file storage unit, 134 Interpolation function storage unit, 136 Key frame data Creation unit, 138 Animation generation unit, 140 Audio file storage unit, 142 Output unit, 160 Phoneme sequence file, 180 Mapping processing unit, 182 Blend rate adjustment unit by duration length, 184 Blend ratio adjustment unit by power, 200 Animation generation control unit, 202 Timer, 204 Interpolation processing unit, 290 Network, 292 Audio input computer, 294 Speech recognition server, 296 Animation display computer, 300 Touch panel, 302 Microphone

Claims (9)

Means for receiving a voice signal and creating key frame data representing a key frame image constituted by an image at a predetermined key frame time during the duration of each phoneme in the phoneme string represented by the voice signal;
An animation generating device, including: animation generating means for generating an animation of an image composed of a series of images that change in synchronization with the audio signal based on the key frame data generated by the key frame data generating means. The animation creating apparatus according to claim 1, wherein the predetermined key frame time is a start time of a duration time of each phoneme in the phoneme string. Text selection means for allowing the user to select a plurality of predetermined types of text;
Means for recording a voice of a user based on the selection of the text by the text selection means, converting the voice into a voice signal, and supplying the voice signal to the key frame data creation means together with the selected text;
The means for creating the key frame data is:
Mapping data storage means for storing mapping data for mapping phonemes to any of a plurality of predetermined images including a predetermined reference image;
Receiving the speech signal and the text, performing phoneme segmentation on the speech signal based on the text, and outputting phoneme sequence data including the obtained phoneme sequence and time information indicating the duration of each phoneme Phoneme segmentation means of
For each phoneme included in the phoneme string data output from the phoneme segmentation means, an identifier for identifying an image to which the phoneme is mapped by referring to the time information of the phoneme and the mapping data And a key frame data creating means for creating and outputting key frame data by attaching a blend ratio determined in correspondence with a predetermined feature amount for the phoneme. Animation creation device. The key frame data creation means includes
For each phoneme included in the phoneme string data output from the phoneme segmentation means, an image identifier obtained by referring to the mapping data and a blending rate consisting of a predetermined constant are attached, and the image mapped Mapping processing means for outputting phoneme sequence data;
First blend rate adjusting means for adjusting the blend rate as a monotonically increasing function of the phoneme duration for each phoneme of the image mapped phoneme string data output by the mapping processing means The animation creating apparatus according to claim 3. The key frame data creation means further monotonously increases the power level within the duration of the phoneme for each phoneme of the phoneme string data adjusted by the blend ratio output from the first blend ratio adjustment means. The animation creating apparatus according to claim 4, further comprising: a second blend ratio adjusting unit that adjusts the blend ratio as a function. The animation generating means includes
Time determination means for determining a generation time for generating an animation image in relation to the recording time of the sound;
Interpolation means for calculating an image of a frame at the generation time determined by the time determination means by interpolation between a plurality of key frame images sandwiching the generation time. An animation creating device according to any one of the above. The interpolation means includes means for calculating an image of a frame at the generation time determined by the time determination means by interpolation between two key frame images adjacent to each other across the generation time. The animation creating apparatus according to claim 6. The means for calculating is
Of the first and second key frames adjacent to each other across the generation time, the generation time is determined by a first interpolation function that is 100% in the first key frame and 0% in the second key frame. First interpolating means for interpolating the first blend rate in the above from the blend rate in the first key frame;
The second blend rate at the generation time is interpolated from the blend rate in the second key frame by a second interpolation function that is 0% in the first key frame and 100% in the second key frame. A second interpolation means for
A weighted sum between the image data mapped to the first key frame and the image data mapped to the second key frame using the first blend rate and the second blend rate. The animation creating apparatus according to claim 7, further comprising: means for calculating image data at the generation time. A computer program that, when executed by a computer, causes the computer to function as each means constituting the animation creation device according to any one of claims 1 to 8.

Images