JP4617500B2 - Lip sync animation creation device, computer program, and face model creation device - Google Patents

Description

  The present invention relates to an animation creating apparatus that creates an animation from voice, and more particularly to an apparatus that automatically generates an animation such as a face image that changes the shape of a mouth or the like in accordance with an uttered voice.

  Due to the development of computer technology, work that has been mostly done manually has been replaced by computer work. A typical example is the creation of animation.

  Previously, animations were generally created by the following method. The animation director decides the characters to appear, and creates a rough original picture of the main scene called a storyboard. Based on these storyboards, a picture of each frame of animation is created by an operator called an animator. The person in charge finishes these pictures into cel drawings. If the cell images are sequentially copied onto the film and played back at a predetermined frame rate, an animation image portion is completed.

  While playing this animation image, the voice actor adds a line based on the script of the animation. This is so-called “post-recording”.

  It is the creation of cel images that requires the most work in such work. On the other hand, when creating an original picture with CG (computer graphics), it is a relatively simple task to create a cell picture by processing the original picture. There is no need to shoot one by one. Therefore, this part is considerably computerized together with the CG conversion of the original picture.

  On the other hand, the remaining task is relatively difficult. Because it is necessary to speak dialogue with the voice of the situation according to the movement of the animation, the post-recording work takes a certain amount of time, and learning is also necessary.

  Therefore, conversely to post-recording, a method of recording audio first and creating an animation according to the audio was considered. This is called "Presco" or "Pre-Reco" (hereinafter referred to as "Presco etc."). This is a technique that was originally used when creating animations manually in the United States. When an animation is created by this method, the work procedure is as follows.

  First, determine the character that will appear in the animation. Create storyboards as before. A voice actor speaks based on a storyboard and script and records it as audio. An animation is created according to this sound.

  When the animation creation by the technique such as Presco is realized by a computer, the problem is how to automatically create the animation from the voice. In particular, it is difficult to generate the movement of the mouth of an animation such as a person in a natural form in accordance with the voice of a voice actor that has been recorded in advance, and there is a need for a method that automatically performs this.

  One method proposed for this purpose is the method described in Patent Document 1. In the method described in Patent Document 1, a plurality of mouth-shaped basic patterns are prepared in advance. Then, a mouth shape corresponding to an arbitrary voice is obtained by a weighted sum of these basic patterns. For this purpose, a conversion function for converting a predetermined feature amount of the voice actor's voice into a weighting parameter of each basic pattern is obtained in advance by multiple regression analysis. The voice actor's voice is recorded into the voice actor's voice by converting the predetermined feature quantity of the voice actor's voice recorded along the script into a weighted parameter using this conversion function and calculating the weighted sum of the basic pattern of the mouth shape using the weighted parameter. Corresponding mouth shape and face image are created. By performing such a process at a time corresponding to each frame of the animation, an animation frame sequence is created.

  FIG. 1 shows an outline of an animation creating process 30 which is a premise of such a conventional animation creating apparatus. Referring to FIG. 1, in the animation creation process 30, when a speaker 40 utters a speech based on a script 44, a phoneme segmentation (a phoneme sequence that constitutes an utterance from the speech) is performed on the speech signal 42. Is generated).

  For main phonemes, face images 60 to 68 including a mouth shape when the phoneme is pronounced are prepared in advance, and these face images are assigned to each phoneme 50 to 58 obtained as a result of speech recognition. To animate.

  In addition, since a smooth image cannot be obtained even if one speech image is assigned to each phoneme, an intermediate image is obtained by a weighted sum between main images as described in Patent Document 1. create. For example, there are five main face images: “A (/ a /)”, “I (/ i /)”, “U (/ u /)”, “E (/ e /)”, “O (/ o /)”. A total of six face images are prepared: five face images for phonemes and face images for phonemes “n / N /”. The face image for “N / N /” is an image that is the basis of another face image as will be described later, and is also referred to as “an expressionless face image” in this specification. The five phonemes “A” to “O” are assigned to the corresponding face images, respectively, and the remaining phonemes are assigned to any of the six face images described above. This is hereinafter referred to as phoneme to face image mapping.

  FIG. 2 shows an example of a face image used. The face image includes an expressionless face image 80 which is the basis of all other face images, and the face images 60 to 68 from “A” to “O” described above. The face images 60 to 68 are generated by pasting a prepared facial texture on the wire frame image. The wire frame images of the face images 60 to 68 and 80 are all defined by the three-dimensional coordinates of each vertex constituting the wire frame. However, the coordinates of each vertex are defined in advance for the basic expressionless facial image 80, but the coordinates of each vertex of the facial images 60 to 68 are defined by relative coordinates with respect to the expressionless facial image 80. A face model composed of a set of coordinates of each vertex constituting the face images 60 to 68 and 80 is hereinafter referred to as a “visual element”.

  In the case of creating an animation based on the face image prepared as described above, the following manual procedure has been conventionally employed. In other words, while listening to the voice, assigning the face image of “A” at the time of uttering the voice of “A” at a certain time, and assigning the face image of “O” when uttering the voice of “O” This is done manually for all such frames that may need to be assigned. A frame to which a face image at the time of uttering a specific voice is assigned is called a “key frame”.

  Next, based on the key frames assigned in this manner, a face image at an arbitrary time point between the key frames is synthesized by blending between the face images assigned to the two key frames sandwiching the time point.

  FIG. 3 shows an example of key frame allocation. In the example shown in FIG. 3, the face image 60 representing “A” is assigned to two key frames as indicated by vertical bars 100 and 102. Similarly, as shown by the vertical bar 110 for the face image 62, by the vertical bar 120 for the face image 64, by the vertical bar 130 for the face image 66, and by the vertical bar 140 for the face image 68, respectively. It is assigned to one frame.

  Face images in these frames (key frames) are created to match the specified face image, but in other frames, they are created by blending between the face images of two key frames that sandwich the frame. Is done. The “weighted sum” referred to in Patent Document 1 is a concept corresponding to this. The graphs 104, 112, 122, 132, and 142 in FIG. 3 represent the blend ratios of the face images 60 to 68, respectively. In the section where the blend rate = 0, the face image is not used for animation creation. In a section where the blend rate is not 0, the face image is created by adding the product of the face image and the blend rate to the product of the other face image and the blend rate.

  The blend ratio is defined as 100% for a specific face image and 0% for the face image / N /, and the intermediate face image at the ratio of the amount of movement of the feature points from the face image / N / to the specific face image. It represents. Accordingly, when the face images / A /, / I /, / U /, / E /, / O / are assigned to phonemes as they are, the blend ratio is 100%. A face image / A / with a blend rate of 50% is a face whose ratio of the amount of movement of feature points from the face image / N / is 50% of the amount of movement of feature points of the face image / A /. Refers to the image. If the movement amount of the feature point of the face image is represented by a vector starting from the position of the face image / N /, the face image with the blend rate B% is a vector representing each feature point, and the direction is the blend rate 100%. This corresponds to a face image that is equal to the vector of the face image and is reduced in length corresponding to the blend ratio B%.

  FIG. 4 shows an example of a face image created by blending in this way. FIG. 4A shows a face image when the blend ratio for the / a / face image is 100%. FIG. 4D shows a face image when the blend ratio for the / i / face image is 100%. FIG. 4B shows a face image at a blending ratio of 65% for / a / and a blending ratio of 35% for / i /, and FIG. 4C shows a blending ratio of 35% for / a /, The face images when the blend ratio of i / is 65% are shown.

  As can be seen from FIGS. 4A to 4D, by changing the blend ratio and blending the two face images on the model to create a new face image, an intermediate face between the two face images is obtained. You can create an image.

Japanese Patent Laid-Open No. 7-44727 Linde Y., Buzo A., Gray R., "An algorithm for vector quantizeer design," IEEE Transactions on Communications. COM-28 (1980), 84-95.

  When an animation of a face image is automatically created by the above-described conventional technique, the problem is where to set a key frame and how to set the blend rate. In the past, humans performed all of them manually, and the resulting animation has a fairly high quality. However, a technique that can automatically set a key frame and a blend ratio thereof and can create a smooth animation similar to a result of human manual work has not been known.

  Setting the key frame and setting the blend rate are the most important and skill-intensive work in creating an animation by blending as described above, and a technique for automating this work is desired.

  Also, unlike movies, animation does not simply mean that smooth images can be obtained. For example, the conventional manual animation has a problem that the movement is awkward because the number of frames per unit time is small. However, there are some people who feel that the weak point is the appeal of animation. Even in lip sync animation, it is desirable to be able to realize movement like manual animation in this way if necessary.

  Furthermore, with the globalization of culture, animations in Japanese are often created in foreign countries, but in the future, animations created in Japanese may be changed for broadcasting in foreign countries. Conventionally, this is realized by so-called dubbing as in the case of a movie. However, in the case of dubbing, the movement of the mouth and the sound do not coincide. With lip sync animation, you can deal well with these problems by first recording the audio and then creating the animation for that audio. However, in that case, it is necessary to prepare resources necessary for animation creation according to the sound used in each language. It is desirable to minimize such preparation work.

  Therefore, an object of the present invention is to create a lip sync animation that can automatically set key frames and blend ratios so that a smooth and natural animation can be obtained when creating an animation of a facial image from voice data of human speech. Is to provide a device.

  Another object of the present invention is to create a key image and its animation so that a smooth and natural animation and an awkward movement animation can be obtained as necessary when creating an animation of a facial image from voice data of human speech. The object is to provide a lip-sync animation creation device capable of automatically setting a blend rate.

  Still another object of the present invention is to create a smooth and natural image while reducing the amount of work as much as possible when creating an animation of a facial image that matches the speech of each language from speech data of multilingual human utterances. It is an object of the present invention to provide a lip-sync animation creation device capable of automatically setting key frames and their blend ratios so that animation can be obtained.

  The lip-sync animation creating apparatus according to the first aspect of the present invention is prepared in advance, corresponding to a statistical acoustic model prepared in advance, mapping definitions between phonemes and visual elements prepared in advance, and visual elements. A lip-sync animation creation apparatus for creating a lip-sync animation from input speech data using a plurality of face models of face images, and transcription for speech data can be used. This lip-sync animation creation device uses statistical acoustic models, mapping definitions, and transcriptions to find phonemes and corresponding visual elements contained in speech data, with a default blend rate and duration Visual element sequence creating means for creating a visual element sequence. A key frame is defined at a predetermined position within the duration of the visual element sequence, and a key frame sequence is defined by a key frame defined within the duration of each visual element of the visual element sequence. The lip sync animation creating device further includes, in order from the fastest change between the face model corresponding to the visual element, between the key frames in the key frame sequence and the adjacent key frames. An animation of a face image is created by blending between key frames based on a key frame deleting means for deleting a predetermined percentage of key frames and a key frame sequence in which some key frames are deleted by the key frame deleting means. Blending processing means.

  The visual element sequence creating means creates a visual element sequence from the speech data using a statistical acoustic model, a mapping definition, and a transcription. This visual element sequence has a duration. A key frame is defined at a predetermined position within the duration of the visual element sequence, and a key frame sequence is defined by a key frame defined within the duration of each visual element of the visual element sequence. Although animation can be created from these keyframes by blending, the resulting animation will be unnatural. Therefore, the key frame deletion means deletes a predetermined percentage of key frames in order from the largest one of the key frames in the key frame sequence with the fastest change of the face model. . By removing the keyframes where the movement is faster, the resulting animation will be natural even when using the default blend ratio. As a result, it is possible to provide a lip sync animation creation device that can automatically set key frames and blend ratios thereof so that a smooth and natural animation can be obtained. This ratio may be adjustable.

  Preferably, the key frame deleting means includes, in the key frames in the key frame sequence, each feature point constituting the face model corresponding to the visual element of the key frame and the face model corresponding to the visual element of the adjacent key frame. , And a means for deleting a predetermined percentage of key frames in order from the one with the highest speed of change between corresponding feature points.

  For each feature point that makes up the face model, calculating the speed of change between adjacent key frames increases the amount of calculation but reduces the error contained in the calculation result, creating a natural animation it can.

  More preferably, the lip-sync animation creation device calculates a motion vector between feature points constituting the face model for all combinations of two face models selected from the plurality of face models. The feature points of the motion vector calculation means and the two face models are clustered by a predetermined clustering method with respect to the motion vector calculated by the motion vector calculation means, and the representative vectors of each cluster are calculated, thereby being clustered. Further comprising means for creating a face model and clustered face model storage means for storing the clustered face model. For each key frame in the key frame sequence, the key frame deletion means generates a clustered face model corresponding to a combination of a visual element of the key frame and a visual element of an adjacent key frame. A moving amount calculating means for calculating a speed of change between key frames of feature points belonging to each cluster using a representative vector of the cluster, and a speed of change calculated by the moving amount calculating means; Means for deleting a predetermined percentage of key frames from the key frame sequence in order from the largest.

  In advance, motion vectors are obtained for all combinations of face models, and each feature point is classified into clusters by predetermined clustering for the motion vectors, for example, vector quantization clustering. The means for creating a clustered face model calculates a representative vector for each cluster. For each key frame in the key frame sequence, the movement amount calculation means calculates a clustered face model corresponding to a combination of a visual element of the key frame and a visual element of an adjacent key frame. Reading from the storage means, the speed of change between key frames of feature points belonging to each cluster is calculated using the representative vector of the cluster. A predetermined percentage of key frames are deleted from the key frame sequence in order from the highest calculated change rate. Instead of calculating the speed of change of each feature point, the feature points belonging to one cluster are represented by one representative point, and the speed of change is calculated, so that the time required for calculation can be shortened.

  More preferably, the lip-sync animation creating device receives a key frame sequence in which some key frames are deleted by the key frame deleting unit, and generates a speech power of a phoneme corresponding to a visual element of the key frame in the key frame sequence. If the average utterance power calculated by the utterance power calculating means for calculating from the utterance data and the duration of the visual element including the key frame is small for each key frame in the key frame sequence by the utterance power calculating means It further includes blend rate adjustment means based on utterance power for adjusting the blend rate by a predetermined function such that the smaller the smaller the blend rate. The blend processing means creates an animation of a face image by blending between key frames based on the key frame sequence whose blend rate is adjusted by the blend rate adjusting means based on speech power.

  Where the utterance power is small, the blend rate is small. In general, when speech power is low, humans do not open their mouths very clearly. Therefore, by doing this, it is possible to realize an animation of a face image that moves close to the speaker's mouth during actual speech. As a result, it is possible to provide a lip sync animation creation device that can automatically set key frames and blend ratios thereof so that a smooth and natural animation can be obtained.

  The lip sync animation creation device receives a key frame sequence in which some key frames have been deleted by the key frame deletion means, receives a vertex constituting the face model corresponding to the visual element of the key frame, and a visual element of the adjacent key frame. Included in the key frame sequence in which some key frames are deleted by the key frame deletion unit and the change rate calculation unit for calculating the change rate between the vertices constituting the face model corresponding to For each key frame, a predetermined function is used such that the blend rate is a smaller value for a key frame whose speed of change calculated by the speed of change calculation means is larger than a predetermined threshold value. And a blend rate adjusting means based on a vertex speed for updating the blend rate. The blend processing means creates an animation of a face image by blending between key frames based on the key frame sequence whose blend rate is adjusted by the blend rate adjusting means based on the vertex speed.

  Preferably, the lip-sync animation creating apparatus calculates a motion vector between feature points constituting a face model for all combinations of two face models selected from a plurality of face models. Clustered faces are obtained by clustering the feature points of the vector calculation means and the two face models by a predetermined clustering method for the motion vector calculated by the motion vector calculation means, and calculating a representative vector of each cluster. Further comprising means for creating a model and clustered face model storage means for storing the clustered face model. The lip sync animation creating device further receives a key frame sequence in which some key frames are deleted by the key frame deleting means, and among each key frame, a visual element of the key frame, a visual element of an adjacent key frame, A clustered face model combination corresponding to each combination is read from the clustered face model storage means, and the speed of change between key frames of feature points belonging to each cluster is calculated using a representative vector of the cluster Of the key frames included in the key frame sequence in which some key frames are deleted by the key frame deletion unit, the change rate calculated by the change rate calculation unit is For key frames that are larger than a given threshold, the blend ratio will be smaller. Further comprising a blend ratio adjustment means according to the vertices rate for updating the blend ratio by using such a predetermined function. The blend processing means creates an animation of a face image by blending between key frames based on the key frame sequence whose blend rate is adjusted by the blend rate adjusting means based on the vertex speed.

  The lip-sync animation creating apparatus according to the second aspect of the present invention includes a statistical acoustic model prepared in advance, mapping definitions between phonemes and visual elements prepared in advance, and a plurality of facial images prepared in advance. A lip-sync animation creation device for creating a lip-sync animation from input utterance data using a facial model, and transcription for the utterance data can be used, statistical acoustic model, mapping definition, And a visual element sequence creating means for obtaining phonemes and corresponding visual elements included in the utterance data by using transcription and creating a visual element sequence having a duration with a default blend ratio. . A key frame is defined at a predetermined position within the duration of the visual element sequence, and a key frame sequence is defined by a key frame defined within the duration of each visual element of the visual element sequence. The lip sync animation creation device further includes speech power calculation means for calculating speech power of phonemes corresponding to visual elements of key frames in the key frame sequence from speech data, and for each key frame in the key frame sequence, Speaking power for adjusting the blending rate by a predetermined function such that the smaller the average utterance power calculated for the duration of the visual element including the key frame by the utterance power calculating means, the smaller the blending rate is. And blend processing means for creating an animation of a face image by blending between key frames based on the visual element sequence whose blend ratio is adjusted by the blend ratio adjusting means.

  Preferably, the lip sync animation creation device receives a key frame sequence whose blend rate is adjusted by the blend rate adjusting means based on speech power, and constructs a face model corresponding to the visual element of each key frame included in the key frame sequence Blending by means of a rate of change calculation for calculating the speed of change between the vertices that make up and the vertices that make up the face model corresponding to the visual element of the adjacent key frame, and a blend rate adjustment unit by utterance power Among the key frames included in the key frame sequence whose rate has been adjusted, the blend rate of the key frame whose speed of change calculated by the speed of change calculation means is greater than a predetermined threshold is more Blur due to vertex velocity to update the blend rate using a predetermined function that is small. Further comprising a de factor adjusting means. The blend processing means creates an animation of a face image by blending between key frames based on the key frame sequence whose blend rate is adjusted by the blend rate adjusting means based on the vertex speed.

  More preferably, the lip-sync animation creation device calculates a motion vector between feature points constituting the face model for all combinations of two face models selected from the plurality of face models. The feature points of the motion vector calculation means and the two face models are clustered by a predetermined clustering method with respect to the motion vector calculated by the motion vector calculation means, and the representative vectors of each cluster are calculated, thereby being clustered. Further comprising means for creating a face model and clustered face model storage means for storing the clustered face model. The lip sync animation creation device further receives a key frame sequence in which the blend rate is adjusted by the speech rate blend rate adjusting means, and among each key frame, the visual element of the key frame, the visual element of the adjacent key frame, A clustered face model combination corresponding to each combination is read from the clustered face model storage means, and the speed of change between key frames of feature points belonging to each cluster is calculated using a representative vector of the cluster The change rate calculation means for each of the key frames included in the key frame sequence and the blend of the key frames whose change speed calculated by the change speed calculation means is greater than a predetermined threshold The top point for updating the blend rate with a predetermined function that gives a smaller value. And a blend ratio adjustment means according to the speed. The blend processing means creates an animation of a face image by blending between key frames based on the key frame sequence whose blend rate is adjusted by the blend rate adjusting means based on the vertex speed.

  A lip-sync animation creating apparatus according to a third aspect of the present invention includes a statistical acoustic model prepared in advance, mapping definitions between phonemes and visual elements prepared in advance, and a plurality of facial images prepared in advance. And a lip sync animation creation device for creating a lip sync animation from input utterance data, and transcription for the utterance data can be used. The lip-sync animation creation device uses a statistical acoustic model, mapping definition, and transcription to determine phonemes and corresponding visual elements contained in speech data, and has a duration with a default blend rate. Visual element sequence creation means for creating a visual element sequence is included. A key frame is defined during the duration of each visual element in the visual element sequence, and a key frame sequence is defined by these key frames. The lip sync animation creating device further includes a vertex that constitutes a face model corresponding to a visual element of each key frame included in the key frame sequence and a vertex that constitutes a face model corresponding to a visual element of an adjacent key frame. A change rate calculation means for calculating the change speed between the key frame sequences, and the change speed calculated by the change speed calculation means among the key frames included in the key frame sequence is a predetermined threshold. For a key frame larger than the value, the blend rate adjustment means based on the vertex speed and the blend rate adjustment means based on the vertex speed for updating the blend ratio using a predetermined function such that the blend ratio becomes a smaller value. Based on the key frame sequence with adjusted blend ratio, the face image is animated by blending between key frames. And a blend processing unit for creating.

  A lip-sync animation creating apparatus according to a fourth aspect of the present invention includes a statistical acoustic model prepared in advance, a mapping definition between phonemes and visual elements prepared in advance, and a plurality of face images prepared in advance. And a lip sync animation creation device for creating a lip sync animation from input utterance data, and transcription for the utterance data can be used. A lip-sync animation creating device is a motion vector calculating means for calculating a motion vector between feature points constituting a face model for all combinations of two face models selected from a plurality of face models. Then, the feature points of the two face models are clustered by a predetermined clustering method for the motion vector calculated by the motion vector calculation means, and a representative vector of each cluster is calculated to create a clustered face model Phonetics included in speech data and correspondence using clustered face model storage means for storing clustered face models, statistical acoustic models, mapping definitions, and transcriptions A keyframe sequence with a continuous length with a default blend rate And a key frame sequence creation means for creating. A key frame is defined during the duration of each visual element in the visual element sequence, and a key frame sequence is defined by these key frames. The lip-sync animation creation device further receives a key frame sequence, and, among each key frame, a clustered face model combination corresponding to a combination of a visual element of the key frame and a visual element of an adjacent key frame. Included in the key frame sequence, which is read from the clustered face model storage means and calculates the speed of change between the key frames of the feature points belonging to each cluster using the representative vector of the cluster, and the key frame sequence Among the key frames to be changed, a predetermined function is set such that the blend rate becomes a smaller value for a key frame whose change speed calculated by the change speed calculation means is larger than a predetermined threshold value. Blend rate adjustment means by vertex speed and blend rate adjustment by vertex speed to update blend rate using Based on the keyframe sequence blend ratio by stages is adjusted, and a blend processing unit for creating animation of the face image by blending between keyframes.

  Preferably, the lip-sync animation creating apparatus includes a key frame duration immediately before a key frame to which a visual element corresponding to a blank phoneme is assigned among key frames included in a key frame sequence output by a visual element sequence creating unit. Utterance end correction means for correcting the utterance end position by moving the end position of the utterance to a position after the maximum point of the utterance power sequence of the utterance data in the key frame and within the duration of the key frame Further included. The key frame deletion means receives as input the key frame sequence whose utterance end has been corrected by the utterance end correction means.

  The end position of the key frame immediately before the key frame to which the visual element corresponding to the blank phoneme is assigned is corrected. The end point after correction is a position after the maximum point of the speech power sequence in the key frame. By moving the corrected end in front of the original end position in this way, the visual element at the end of the utterance becomes a visual element corresponding to the blank phoneme earlier, and the utterance animation becomes natural.

  More preferably, the utterance end correction means includes the utterance of the key frame immediately before the key frame to which the visual element corresponding to the blank phoneme is assigned among the key frames included in the key frame sequence output by the visual element sequence generation means. Means for detecting a first time for giving a maximum value of power, and speech power at a predetermined rate from the maximum value of speech power after the first time and before the end time of the key frame to be processed. Means for detecting a decreasing second time, and means for correcting the key frame so as to move the end position of the key frame to be processed to the second time.

  After the first time when the maximum value of the utterance power is given, the end position of the key frame is moved to a second time when the utterance power decreases by a predetermined rate from the maximum value. Regardless of the magnitude of the absolute value of the utterance power in each key frame, the movement position of the end position is determined by the attenuation rate from the maximum value, so that it is stable at the end of the utterance regardless of the fluctuation of the utterance power magnitude. An image that closes the mouth at the timing is obtained.

  More preferably, the key frame creation means selects any one of the frames at the first frame rate as the key frame when creating the key frame sequence. The lip sync animation creating apparatus is further connected to receive an input designating a second frame rate smaller than the first frame rate and a key frame sequence output by the key frame deleting means, and the key frame deleting means Includes a frame rate converting means for converting the key frame sequence output by the key frame sequence into a key frame sequence of the second frame rate. The frame rate conversion means is assigned to each key frame of the key frame sequence of the second frame rate, in the key frame sequence output by the key frame deletion means, to a key frame having a start edge within the duration of the key frame. Assign one of the visual elements. The blend processing means includes means for creating an animation of a face image by blending between key frames based on the key frame sequence whose frame rate has been converted by the frame rate conversion means.

  The key frame creation means creates a key frame sequence using an arbitrary frame among the frames at the first frame rate. When a second frame rate smaller than the first frame rate is designated, the key frame rate conversion means converts the key frame sequence having the first frame rate into a key frame sequence having the second frame rate. At this time, there is a possibility that a plurality of key frames in the key frame sequence at the first frame rate correspond to key frames in the key frame sequence at the second frame rate. In such a case, the frame rate converting means has a key frame visual element of the key frame sequence of the first key frame rate having a start edge within the duration of the key frame in the key frame sequence of the second key frame rate. Is assigned to the converted key frame. Since a key element in the key frame sequence at the second key frame rate is always assigned a visual element of a key frame having a start point within the duration of the key frame, it follows the visual element before the actual voice is spoken. Mouth shape changes will begin. Since this order matches the order observed during actual human speech, a face image animation with natural speech is obtained.

  The frame rate conversion means may assign the visual element so that the visual element assigned to each key frame of the key frame sequence of the second frame rate is different from the visual element assigned to the immediately preceding key frame. .

  If key frames to which the same visual element is assigned continue, the same mouth shape will continue for a long time, making it unnatural as a face image during speech. By assigning a visual element different from the visual element assigned to the immediately preceding key frame to each key frame, such unnaturalness can be avoided and a more natural facial image animation can be created.

  More preferably, the blend processing means has a function of creating an image for each frame at a third frame rate higher than the second key frame rate when creating an animation from the key frame sequence of the second frame rate. And having a function of generating an image of a frame between the adjacent key frames by interpolation between the adjacent key frames. The lip sync animation creating apparatus further includes, for each key frame in the key frame sequence of the second frame rate output from the frame rate conversion means, a frame between the key frame and the key frame immediately after the key frame. The position includes a key frame copy means for copying the same key frame as the key frame.

  More preferably, the key frame copy means, for each key frame in the key frame sequence of the second frame rate output from the frame rate conversion means, at the frame position immediately before the key frame immediately after the key frame, Means for copying the same keyframe as the keyframe.

  The blend processing means creates a frame according to the third frame rate between two adjacent key frames of the second frame rate, and the image in those frames is converted to the two key frames. When creating by interpolation between frames, a frame according to a third frame rate that smoothly changes is inserted between two key frames. When such an interpolation process is performed, the change in the image becomes smooth, but a video with a “limit feeling” sometimes required for animation (a video that changes as “cracking”) cannot be obtained. In that case, the former key frame is copied as it is to the frame position immediately before the latter of the two adjacent key frames. As a result, from the former key frame position to the copied frame position, even if interpolation is performed by the blend processing means, the image is stable and does not change, and the image changes only at the next key frame immediately after that. Become. As a result, when the frame sequence is created according to the third frame rate larger than the second frame rate, the blend processing means having the function of creating the frame image between the adjacent key frames is used as it is. Even in this case, you can create an animation with a sense of limit.

  More preferably, the lip sync animation creating apparatus further includes a face model storage means for storing a face model of a plurality of face images.

  Face models of a plurality of face images can be stored by the face model storage means. Even when the animation is repeatedly created, the same face model can be used any number of times without repeatedly receiving the face model from the outside.

  More preferably, the phoneme prepared in advance includes a predetermined standard phoneme and a general phoneme other than the standard phoneme, and the face model of the plurality of face images includes a standard visual composed of a face model corresponding to the standard phoneme. And a general visual element model including a face model corresponding to a general phoneme. The lip sync animation creation device further captures the measured values of the three-dimensional positions of the feature points of the face image of the speaker when the corresponding phoneme is spoken, which is classified in advance corresponding to the phonemes prepared in advance. General visual element generation means for generating a general visual element model using the data and the standard visual element model is included.

  If only the standard visual element model is created in advance by hand, and the capture data of the face of the actual speaker at the time of utterance is prepared, the general visual element creation means uses the general visual element creation means to generate general visuals other than the standard visual element model. Generate an elementary model automatically. Therefore, the amount of work for creating a face model by manual work is reduced, and a smoother and more natural face image animation in which the movement of the mouth and the sound coincide with each other can be obtained.

  More preferably, the general visual element generating means uses a linear sum of the capture data corresponding to the standard phonemes, and calculates a predetermined approximation error with the same number of coefficients as the number of standard phonemes for approximating the capture data corresponding to the general phonemes. And a linear sum of a standard visual element model using a coefficient calculated by the coefficient calculating means for a general phoneme corresponding to the general visual element model. And a linear sum calculation means for storing in the face model storage means in association with the corresponding general phoneme together with the standard visual element model.

  The apparatus generates a general visual element model with a linear sum of standard visual element models that minimize the approximation error. Since a face image for each phoneme can be generated using not only the standard visual element model but also the general visual element model, a smooth and natural face image animation can be obtained.

  When the computer program according to the fifth aspect of the present invention is executed by a computer, it causes the computer to function as any one of the above-described lip-sync animation creating apparatuses.

  A face model generation apparatus according to a sixth aspect of the present invention uses a mapping definition between phonemes and visual elements prepared in advance, and generates a face model of a face image corresponding to the visual elements. The phonemes prepared in advance include predetermined standard phonemes and general phonemes other than the standard phonemes, and the face models of the plurality of face images are standard visions composed of face models corresponding to the standard phonemes. A face model storage unit for storing a face model of a plurality of face images corresponding to visual elements; and a general visual element model including a face model corresponding to a general phoneme. Capture data and standard visual element model, which are pre-classified in correspondence with phonemes prepared in advance, and are composed of measured values of three-dimensional positions of feature points of the face image of the speaker when speaking the corresponding phoneme. Used, general And a general visual element generating means for generating a Satoshimoto model.

  Preferably, the general visual element generating means is a linear sum of the capture data corresponding to the standard phonemes, and has the same number of coefficients as the number of standard phonemes for approximating the capture data corresponding to the general phonemes, and a predetermined approximation error. Coefficient calculation means for calculating to be minimized, and a general visual element model by linear sum of standard visual element models using coefficients calculated by coefficient calculation means for general phonemes corresponding to the general visual element model. Linear sum calculation means for calculating and storing in the face model storage means in association with the corresponding general phoneme together with the standard visual element model.

  Hereinafter, the present invention will be described based on embodiments. In the following description, six types of basic face images are used, but the number of face images is not limited to this. There may be fewer than six types or more than six types.

[First Embodiment]
<Configuration>

  FIG. 5 shows a schematic block diagram of a lip sync animation creating apparatus 200 according to the first embodiment of the present invention as an example of the animation creating apparatus according to the present invention. Referring to FIG. 5, lip-sync animation creating apparatus 200 includes speech data stored in utterance storage unit 152 and utterance stored in utterance storage unit 152 stored in transcription storage unit 154. 3D character model corresponding to six basic face images composed of / a / ˜ / o / and / N / stored in the character model storage unit 156, receiving a transcription text (transcription) as an input Is used to create an animation 260 of a face image.

  An example of a face image stored in the character model storage unit 156 is shown in FIG. FIGS. 7A to 7F are face images corresponding to phonemes / a /, / i /, / u /, / n /, / e /, / o /, respectively. In this specification, these images are expressed as face images / A /, / I /, / U /, / N /, / E /, and / O /, respectively.

  In the present embodiment, the face images / A /, / I /, / U /, / E /, / O / are all based on the face image / N /, and each feature point represents the face image. In the defined three-dimensional space, it is defined by three-dimensional vector information indicating how far the corresponding feature point of the face image / N / has moved. Accordingly, for example, an intermediate face image can be defined between the face image / A / and the face image / N /. In this embodiment, in order to define an intermediate face image between a specific face image and the face image / N /, the concept of “blend rate” described above is used.

  The blending between the two face images is as described above.

  The lip-sync animation creating apparatus 200 stores an acoustic model storage unit 170 for storing an acoustic model created in advance from the voice of a speaker, and a mapping table between phonemes and visual elements prepared in advance. Using the phoneme-visual element mapping table storage unit 176, the acoustic model stored in the acoustic model storage unit 170, and the phoneme-visual element mapping table stored in the phoneme-visual element mapping table storage unit 176. The phoneme segmentation based on the transcription stored in the transcription storage unit 154 is performed to create a phoneme sequence, and each phoneme in the obtained phoneme sequence is stored in the phoneme-visual element mapping table storage unit 176. Pair using stored phoneme-visual element mapping table The visual element sequence creating unit 230 for creating a visual element sequence with a continuous length by converting the visual element sequence into a visual element, and a visual element sequence storage for storing the visual element sequence output by the visual element sequence creating unit 230 Part 180. The duration of a visual element starts from the beginning of the corresponding phoneme duration. Therefore, among the visual element sequences stored in the visual element sequence storage unit 180, the first frame of each visual element is a key frame. A key frame sequence is composed of key frames in the visual elementary sequence. Note that the visual element sequence creation unit 230 creates a visual element sequence by attaching a phoneme before replacement and a default blend ratio (for example, 100%) to each visual element.

  The lip sync animation creating apparatus 200 further uses a VQ (vector) using a motion vector between any two face images for the vertices constituting each face image of the 3D character model stored in the character model storage unit 156. (Quantization) Clustering is performed, and motion vector data representing the motion of each vertex between any two face images by the representative vector of the cluster to which each vertex belongs, and the model of the face image after clustering at that time A clustering processing unit 232 for outputting and a clustered face model storage unit 234 for storing a face image model after clustering and motion vector data for any combination of face image models output by the clustering processing unit 232 And a face image model stored in the character model storage unit 156 Using one of the clustered face model and motion vector data stored in the clustered face model storage unit 234, a key frame having a fast vertex motion is detected, and such a key frame is detected. And a key frame deleting unit 236 for deleting a predetermined ratio. In this embodiment, when a certain key frame is deleted, the portion that was the continuation length of the key frame is integrated with the continuation length of the key frame immediately before the deleted key frame.

  The lip sync animation creation apparatus 200 further includes a deletion rate input unit for designating a ratio indicated by the number of key frames to be deleted out of the total number of key frames when the key frame deletion unit 236 deletes the key frames. 201 and the motion after clustering stored in the clustered face model storage unit 234 are used as they are for the speed calculation at the time of key frame deletion by the key frame deletion unit 236 or the model stored in the character model storage unit 156 And a cluster processing designation unit 202 for designating whether to use a vector model. Details of the key frame deletion unit 236 will be described later.

  The lip sync animation creating apparatus 200 further includes an utterance power calculation unit 238 for calculating the utterance power in each frame from the utterance data stored in the utterance storage unit 152, and the utterance power calculated by the utterance power calculation unit 238. For the visual element sequence output by the utterance power storage unit 240 for storing and the key frame deletion unit 236, as described later, based on the utterance power in each frame stored in the utterance power storage unit 240, the key And a blend rate adjustment unit 244 based on speech power for adjusting the blend rate of the frame.

  The lip sync animation creating apparatus 200 further allows the user to input a parameter α (hereinafter referred to as “attenuation rate α”) for attenuating the blend rate of a certain key frame in the blend rate adjustment unit 244 based on speech power. Attenuation rate input unit 206, speech power use instruction input unit 204 used when the user instructs whether or not to adjust the blend rate by speech rate blend rate adjustment unit 244, and speech power use instruction input unit 204 When the utterance power is instructed, the output of the key frame deletion unit 236 is given to the blend rate adjustment unit 244 based on the utterance power; otherwise, the output of the key frame deletion unit 236 is bypassed the blend rate adjustment unit 244 based on the utterance power. A pair of selectors 242 and 24 for feeding to subsequent processing units. 6 and so on.

  The lip sync animation creating apparatus 200 further includes the face image model data stored in the character model storage unit 156 and the motion vector stored in the clustered face model storage unit 234 according to the values specified by the cluster processing specification unit 202. To calculate the speed of vertex movement at each key frame, and blend by vertex speed to adjust the blend rate to be smaller for key frames whose movement speed is larger than a predetermined reference The rate adjustment unit 250, the attenuation rate input unit 210 used by the user to input the blend rate attenuation rate β when the blend rate adjustment unit 250 adjusts the blend rate, and the blend rate by the blend rate adjustment unit 250 A vertex speed use instruction input unit 208 for the user to specify whether or not to perform adjustment, A pair that selects whether to output the output of the selection unit 246 to the blend rate adjustment unit 250 or to bypass the blend rate adjustment unit 244 based on speech power and give it to the subsequent processing unit according to the instruction input by the instruction input unit 208. Selection units 248 and 252.

  The lip-sync animation creating apparatus 200 further stores a visual element sequence storage unit 254 for storing a visual element sequence with a continuous length, which is output from the selection unit 252 and for which the blend ratio has been adjusted, and a visual element sequence storage unit 254. A blend processing unit 256 for creating an animation 260 of the face image by performing blend processing using each face image model stored in the character model storage unit 156 based on the visual element sequence with duration. .

  FIG. 6 shows a detailed configuration of the visual element sequence creation unit 230 of FIG. With reference to FIG. 6, the visual element sequence creation unit 230 uses the acoustic model stored in the acoustic model storage unit 170 and uses the acoustic model stored in the speech storage unit 152 to store the transcription in the transcription storage unit 154. Perform phoneme segmentation based on stored transcription, store phoneme segmentation unit 172 for outputting phoneme sequence together with information indicating its duration, and store phoneme sequence with duration length output from phoneme segmentation unit 172 A phoneme sequence storage unit 174.

  The visual element sequence creation unit 230 further includes a phoneme-visual element mapping table storage unit 176 for storing a mapping table between phonemes and visual elements, and a phoneme stored in the phoneme-visual element mapping table storage unit 176. While referring to the visual element mapping table, by converting each phoneme in the phoneme sequence stored in the phoneme sequence storage unit 174 into a corresponding visual element, a phoneme-visual element for outputting a visual element sequence with a duration is output. A conversion processing unit 178. As described above, each visual element of the visual element sequence with a continuous length output from the phoneme-visual element conversion processing unit 178 is assigned a corresponding phoneme and a default blend rate.

  The phoneme segmentation unit 172 may be anything as long as it can perform phoneme segmentation on the utterance data included in the utterance storage unit 152 and output a phoneme string and time data in which each duration is known. Since the utterance content is known in advance by the transcription stored in the transcription storage unit 154, the phoneme segmentation unit 172 can convert the voice data into a phoneme string with high accuracy.

  Table 1 shows a part of an example of the mapping table stored in the mapping table storage unit 176.

テーブル１を参照して、本実施の形態では、マッピングテーブルは、音素／ａ／を視覚素／Ａ／に、音素／ｉ／を視覚素／Ｉ／に、音素／ｕ／を視覚素／Ｕ／に、音素／ｅ／を視覚素／Ｅ／に、音素／ｏ／を視覚素／Ｏ／にそれぞれ対応付けている。マッピングテーブルでは、図３に示す顔画像／Ａ／，／Ｉ／，／Ｕ／，／Ｅ／，／Ｏ／のように、予めある音素に対して準備された視覚素には、その音素を必ず対応付けるようにする。さもないと得られる顔の動画像が発話内容とちぐはぐになってしまう。また音素／Ｎ／、／ｐ／、／ｂ／、／ｍ／等、唇を閉じるような音素は無表情の顔画像／Ｎ／に対応付ける。音素／ｈ／、／ｊ／、／ｑ／、／ｒ／については無視し、視覚素に変換しない。第１のテーブルに記載された音素以外の音素については、直前の音素のブレンド率の８０％のブレンド率を割当てる。

Referring to Table 1, in this embodiment, the mapping table includes phonemes / a / to visual elements / A /, phonemes / i / to visual elements / I /, and phonemes / u / to visual elements / U. /, Phoneme / e / is associated with visual element / E /, and phoneme / o / is associated with visual element / O /. In the mapping table, a phoneme prepared for a certain phoneme in advance, such as face images / A /, / I /, / U /, / E /, / O / shown in FIG. Make sure to associate them. Otherwise, the obtained moving image of the face will be inconsistent with the utterance content. Phonemes that close the lips, such as phonemes / N /, / p /, / b /, / m /, etc., are associated with an expressionless facial image / N /. Phonemes / h /, / j /, / q /, / r / are ignored and are not converted to visual elements. For phonemes other than the phonemes listed in the first table, a blend rate of 80% of the blend rate of the previous phoneme is assigned.

  The processing by the clustering processing unit 232 will be described with reference to FIGS. The processing by the clustering processing unit 232 is simply as follows.

  The following processing is performed for all two arbitrary combinations of the face models included in the character model storage unit 156.

  First, the coordinate vector of all the vertices of one face image is subtracted from the coordinate vector of the other corresponding vertex. By this subtraction, the motion vector of each vertex when changing from one face image to the other face image is obtained. FIG. 8 shows an example of a face image 280 made up of the vertices of visual elements / N / as one face image and a face image 282 made up of the vertices of visual elements / O / as the other face image. An image 284 comprising a set of motion vectors from / to a visual element / O / is shown. In FIG. 8, the horizontal axis is the X axis, the vertical axis is the Z axis, and the Y axis is not shown.

  The clustering processing unit 232 performs clustering on the set of motion vectors thus obtained by the following algorithm.

  (1) The number N of clusters is determined.

  (2) N vectors are arbitrarily selected from the set of motion vectors, and set as an initial codebook.

  (3) All vectors in the set of motion vectors are classified into N clusters based on the Euclidean distance from the initial codebook. In this case, each motion vector is classified into a cluster represented by a code book having the smallest Euclidean distance.

  (4) New N codebooks are created by calculating the average of the vectors belonging to each cluster.

  (5) Repeat steps 3 and 4 until the codebook no longer changes or the difference between them is less than the threshold.

  In the present embodiment, the representative vertex of each cluster is the vertex closest to the centroid (center of gravity) obtained for that cluster.

  As a result of the clustering obtained as described above, each vertex belongs to one of a plurality of clusters for each image combination. FIG. 9 shows an example in which the result of such clustering is mapped to a face image. Referring to FIG. 9, as a result of clustering motion vectors between image 300 and another image (not shown), each vertex constituting the face model constituting image 300 is clustered as shown in image 302. 310, 312, 314, 316, 318, 320, 322 and 324. In this example, the number of clusters is 8, and the number of vertices is 1483.

  As can be seen from FIG. 9, the vertices near the mouth are clearly clustered according to their positions, but there is not much difference in the movement of the vertices in other regions.

  FIG. 10 shows a result 340 of mapping a cluster obtained by clustering in the same process when the number of clusters is 128 and the number of vertices is 1483 to the face image. It can be seen that when the number of clusters is increased in this way, each vertex other than the vicinity of the mouth is also clustered.

  The reason for clustering in this way is as follows. For example, in the processing in the key frame deletion unit 236 and the blend rate adjustment unit 250 shown in FIG. 5, if the movement amount or speed is calculated for all the vertices, it is necessary to calculate the number of vertices, and the processing takes a long time. On the other hand, when vertices are clustered, the movement amount or speed of each vertex can be approximated by the movement amount or speed of the representative vertex of the cluster to which the vertex belongs. Therefore, the substantial calculation amount is reduced to the number of clusters, and the calculation time can be greatly shortened.

  For example, if it is necessary to process only the image near the mouth in a short time, the number of clusters is reduced, and even if the calculation time is somewhat long, it is necessary to obtain not only the mouth but also the entire head image with a certain degree of precision. The number of clusters should be increased. Furthermore, if the time required for the calculation is not limited, such movement amount or speed may be calculated individually for all the vertices without performing such clustering.

  FIG. 11 is a flowchart showing a control structure of a program when the function of the key frame deletion unit 236 is realized by a computer program. Referring to FIG. 11, in step 360, the deletion rate is read from a predetermined storage area. This deletion rate is input in advance by the user using the deletion rate input unit 201 shown in FIG. 5 and stored in a predetermined storage area.

  In step 362, processing for calculating the number K of key frames to be deleted is performed based on this deletion rate. Assuming that the number of key frames in the visual element sequence stored in the visual element sequence storage unit 180 is a and the deletion rate is γ%, in this embodiment, the number K of key frames to be deleted is obtained by a × γ × 100. . Here, whether the calculation result is rounded, rounded up or rounded down is a design matter.

  In step 364, 0 is assigned to the iteration variable i for the following iteration process. In step 366, 1 is added to the variable i. In step 368, it is determined whether or not the value of the variable i is larger than the number K of key frames to be deleted. If the determination result is YES, the process proceeds to step 382, and otherwise, the process proceeds to step 370.

  In Step 370, the clustered face model storage unit 234 uses the clustered face image model stored in the clustered face model storage unit 234 or the original face image model stored in the character model storage unit 156 in the following calculation. Judge whether to do. This determination is performed based on information that is input in advance by the user using the cluster process designating unit 202 and stored in a predetermined storage area. When the clustered model is used, the process proceeds to step 376, and when not used, the process proceeds to step 372.

  In step 372, the distance D between the key frames is calculated by the following equation using all the vertices in all combinations of adjacent key frames in the visual element sequence.

ここで、Ｄ（ｋ）はｋ番目のキーフレームと、ｋ＋１番目のキーフレームとの間の全頂点のユークリッド距離の合計を表す。この距離Ｄ（ｋ）を、以後ｋ番目のキーフレームとｋ＋１番目のキーフレームとの間のキーフレーム間の距離と呼ぶ。

Here, D (k) represents the sum of the Euclidean distances of all vertices between the k-th key frame and the (k + 1) -th key frame. This distance D (k) is hereinafter referred to as a distance between key frames between the k-th key frame and the k + 1-th key frame.

  Subsequently, in step 374, based on the distance between the key frames calculated in step 372, a key frame to be deleted is determined by the following equation.

ただしＤｕｒ_ｋはｋ番目のキーフレームの継続長を示す。

However Dur _k denotes the duration of the k-th key frame.

  In short, as a result of the processing in step 372 and step 374, the key frame having the largest sum of the moving speed of all the vertices from the previous key frame and the moving speed of all the vertices up to the next key frame is obtained. It is determined as a key frame to be deleted. In step 380, the key frame is deleted, and the process returns to step 366.

  On the other hand, if it is determined in step 370 that the model after clustering is to be used, in step 376, the representative vertices of each cluster are used in all the combinations of adjacent key frames in the visual element sequence according to the following formula. Then, the distance D ′ between the key frames is calculated by the following equation.

ただしｍ_ｒは代表頂点ｒにより代表されるクラスタに属する頂点の数を示す。

Here, _mr represents the number of vertices belonging to the cluster represented by the representative vertex r.

  In step 378, based on the distance D 'between key frames calculated in step 376, a key frame to be deleted is determined by the following equation.

要するに、ステップ３７６及び３７８の処理により、キーフレーム間の全ての頂点の移動速度を、代表頂点の移動速度で近似し、それらを用いて一つ前及び一つ後のキーフレームの間の頂点の移動速度の合計が最も大きなキーフレームが削除対象のキーフレームとして決定される。ステップ３８０でこのキーフレームを削除し、ステップ３６６に戻る。

In short, the movement speed of all the vertices between key frames is approximated by the movement speed of the representative vertices by the processing in steps 376 and 378, and the vertices between the previous and next key frames are used by using them. The key frame having the largest moving speed is determined as the key frame to be deleted. In step 380, the key frame is deleted, and the process returns to step 366.

  The processing in step 372 needs to be performed for all vertices constituting the face image model. On the other hand, the processing in step 376 may be performed only on the representative vertex of each cluster. Therefore, the time required for the process in step 376 is much less than the time required for the process in step 372. However, the distance D ′ obtained in step 376 is an approximate value as compared with the distance D obtained in the process in step 372, and includes an error. In some cases, the key frame to be deleted differs between the two.

  If it is determined in step 368 that the value of the variable i is greater than the number K of deleted frames, in step 382, the visual element sequence after the K key frames have been deleted is output, and the process ends.

  FIG. 12 shows the concept of key frame deletion performed by the key frame deletion unit 236. Referring to FIG. 12A, it is assumed that there are four key frames 400, 402, 404, and 406 in the visual element sequence. The distance D or D ′ described above is calculated for all these combinations. Of these, the key frame that gives the minimum value as the total value of the moving speeds of the vertices between the previous and next key frames is deleted. In the example shown in FIG. 12A, it is assumed that the key frame 402 is such a key frame. Then, as shown in FIG. 12B, the key frame 402 is deleted from the visual element sequence, and the above-described processing is performed on the visual element sequence newly including three visual elements.

  Processing performed by the blend rate adjustment unit 244 based on speech power shown in FIG. 5 will be described with reference to FIG. The utterance power blend rate adjustment unit 244 converts the utterance power over the phoneme duration corresponding to each key frame into the utterance data stored in the utterance storage unit 152 and the visual element sequence stored in the visual element sequence storage unit 180. Calculated from the continuation length of the included phoneme sequence. The utterance power of a phoneme is determined by the sum of squares of the amplitude of the speech signal at the center of the duration of each phoneme.

  For example, as shown in FIG. 13, the waveform of the actual audio signal is shown by a graph 420, and the phonemes / a /, / i /, / o /, / e are included in the audio signal shown by the graph 420. Assume that there is a phoneme sequence consisting of / and / u /. For phonemes / a /, the average speech power over the period from the beginning of the duration to the next key frame / i / is calculated. The same applies to the other phonemes / i /, / o /, / e /, and / u /. The average speech power over the period from the beginning of each duration to the next key frame is Calculate over their duration as indicated by minutes 430, 432, 434, 436 and 438. The blend rate adjustment unit 244 based on speech power adjusts the blend rate of visual elements corresponding to each phoneme based on the average value of the speech power thus calculated.

  FIG. 14 is a flowchart showing the control structure of the program when the processing performed by the blend rate adjustment unit 244 based on speech power is realized by a computer program.

  Referring to FIG. 14, in step 450, attenuation factor α is read from a predetermined storage area. The attenuation rate α is input by the user using the attenuation rate input unit 206 shown in FIG. 5 and stored in a predetermined storage area.

  In step 452, the average speech power over the duration is calculated for all phonemes in the phoneme sequence. Hereinafter, the average of the utterance power of the phoneme of the Nth key frame over the entire duration is written as SP (N).

  In step 454, the maximum MAX (SP) and the minimum MIN (SP) are determined among the average values of all the utterance powers calculated in step 452.

  In step 456, the blend rate is updated according to the following equation for all key frames except the key frame that gives the maximum value of the average speech power. Hereinafter, the blend ratio of the Nth key frame is written as BR (N).

平均発話パワーの最大値を与えるキーフレームを除く全てのキーフレームに対してこの式によるブレンド率の調整を行なうと、発話パワーによるブレンド率調整部２４４による処理は終了する。なお、減衰率αは、最小値を与えるキーフレームのブレンド率をどの程度減衰させるかを表していることが上の式から分かる。

When the blend rate is adjusted for all key frames except the key frame that gives the maximum value of the average utterance power, the processing by the blend rate adjustment unit 244 based on the utterance power ends. It can be seen from the above formula that the attenuation rate α represents how much the blend rate of the key frame that gives the minimum value is attenuated.

  An example of the result of this process is shown in the following table. Table 2 shows the blend rate before adjustment and the average utterance power for the phonemes of all key frames, and Table 3 shows the blend rate after adjustment by the blend rate adjustment unit 244 based on utterance power. is there.

ブレンド率に対しこのような調整を行なうことにより、平均発話パワーが最大となるキーフレームのブレンド率は変化しないが、平均の発話パワーが小さくなればなる程、ブレンド率が小さくなる。その結果、話し声が小さい場合には口の動きも小さくなるアニメーションが作成でき、アニメーションの動きがより自然に近くなる。

By making such an adjustment to the blend rate, the blend rate of the key frame that maximizes the average speech power does not change, but the blend rate decreases as the average speech power decreases. As a result, when the speaking voice is low, an animation can be created in which the movement of the mouth becomes small, and the movement of the animation becomes more natural.

  FIG. 15 is a flowchart showing a program control structure when the processing performed by the blend rate adjustment unit 250 of FIG. 5 is realized by a computer program.

  Referring to FIG. 15, in step 470, attenuation factor β is read from a predetermined storage area. The attenuation rate β is input by the user using the attenuation rate input unit 210 shown in FIG. 5 and stored in a predetermined storage area. The meaning of the attenuation rate β will be apparent from the following, but in this embodiment, the proportion of key frames (hereinafter referred to as “invariant frames”) in which the blend rate is not adjusted based on the movement of the vertices between key frames. The indicated value is used.

  In step 472, the number of invariant frames L is calculated by multiplying the total number of key frames by the attenuation rate β read in step 470. It is a design matter whether the number L of invariant frames is obtained by rounding up, rounding off, or rounding down.

  In step 474, it is determined whether to use the model after clustering. This determination is performed by using a value input by the user using the cluster process designating unit 202 and stored in a predetermined storage area. If the model after clustering is used, the process proceeds to step 480, and if not used, the process proceeds to step 476.

  In step 476, the average speed of all the vertices between all key frames and the preceding and following key frames is calculated. This calculation method is the same as that performed in steps 372 and 374 of FIG.

  In step 478, all key frames are sorted in descending order of the average speed calculated in step 476.

  In step 484, the maximum value <VS> of the average speed in the L key frames from the lower order among the data of the key frames sorted in this way is determined.

  In step 486, the blend rate BR (N) is adjusted according to the following equation in a key frame having an average speed larger than the value <VS> determined in step 484.

ただしＶＳ（Ｎ）はＮ番目のキーフレームの平均速度である。ステップ４８６の後、処理を終了する。

Where VS (N) is the average speed of the Nth key frame. After step 486, the process ends.

  On the other hand, when the clustered model is used, in step 480, the average speed of the vertices between all the key frames and the preceding and following key frames is calculated using the representative vertices of each cluster. This processing is performed based on the same concept as that performed in steps 376 and 378 in FIG.

  In step 482, all key frames are sorted in descending order of the average speed calculated in step 480. Thereafter, the process proceeds to step 484.

  In short, the processing here is to adjust the blend rate so that the movement of the mouth becomes small with respect to the speed of the other key frames for the key frame where the moving speed of each vertex is fast. If the movement of the vertices is too fast between key frames, keeping the mouth shape at the key frames intact, the mouth movements look unnatural. Therefore, in such a case, the movement of the mouth is reduced by adjusting the blend rate to be small.

  The following table shows an example of a change in the blend rate before and after the blend rate adjustment unit 250 adjusts the blend rate. Table 4 shows after adjusting the average speed before sorting key frames, and Table 5 shows after sorting and before adjusting blend ratio.

ここで、減衰率β＝６０％とすると、不変フレーム数Ｌは５×０．６＝３となる。したがって表×における下３行についてはブレンド率の調整は行なわず、上２行のみのブレンド率の調整を行なう。ステップ４８４で決定する平均速度の最大値＜ＶＳ＞は、音素／ａ／の平均速度「１００」となる。

Here, if the attenuation rate β = 60%, the invariant frame number L is 5 × 0.6 = 3. Therefore, the blend rate is not adjusted for the lower three rows in the table x, and the blend rate is adjusted only for the upper two rows. The maximum value <VS> of the average speed determined in step 484 is the average speed “100” of phonemes / a /.

  When the processing of step 486 is performed using <VS> = 100, the blend ratios of the upper two phonemes / i / and / o / are corrected as follows. That is, BR (N) = 90 × 100/200 = 45 for phonemes / i / and BR (N) = 60 × 100/150 = 40 for phonemes / o /. As a result, the blend rate of each key frame after the blend rate adjustment by the blend rate adjustment unit 250 is as follows.

すなわち、不変フレームの中の最大の平均速度より大きな平均速度を持つキーフレームのブレンド率が当初より小さな値に調整される。しかも、そのキーフレームの平均速度が大きいほど、ブレンド率は小さくなるため、キーフレームの頂点の移動速度が速いほど、そのキーフレームにおける口の位置の変化が小さくなり、一連のアニメーションはより滑らかで自然なものとなる。

That is, the blend rate of key frames having an average speed larger than the maximum average speed in the invariant frame is adjusted to a value smaller than the initial value. Moreover, since the blend rate decreases as the average speed of the key frame increases, the change in the mouth position at the key frame decreases as the moving speed of the key frame vertex increases, and the series of animations becomes smoother. It will be natural.

<Operation>
The lip sync animation creating apparatus 200 described above operates as follows. Referring to FIG. 5, firstly, utterance data in which an utterance of a predetermined speaker is recorded is prepared in utterance storage unit 152, and transcription that is transcription data of the utterance is prepared in transcription storage unit 154. Is done. In addition, character models of six face images corresponding to the six visual elements described above are prepared in the character model storage unit 156 as wire frame images.

  In order to create the animation 260 of the face image, various preparation operations are necessary. These preparatory tasks are described in turn below.

-Creation of visual elementary sequences-
First, the visual element sequence creation unit 230 uses the acoustic model stored in the acoustic model storage unit 170 and the phoneme-visual element mapping table storage unit 176 stored in the phoneme-visual element mapping table storage unit 176 as follows. Thus, a visual element sequence is created and stored in the visual element sequence storage unit 180.

  Referring to FIG. 6, phoneme segmentation unit 172 of visual element sequence creation unit 230 reads the utterance data in utterance storage unit 152 and uses the transcription storage unit 154 and acoustic model storage unit 170 to process the utterance data. Perform phoneme segmentation. As a result of this processing, the phoneme segmentation unit 172 outputs a phoneme sequence together with data representing the duration of each phoneme. This phoneme sequence with continuous length is stored in the phoneme sequence storage unit 174.

  The phoneme-visual element conversion processing unit 178 reads the phoneme sequence from the phoneme sequence storage unit 174, and uses the phoneme-visual element mapping table stored in the phoneme-visual element mapping table storage unit 176 to convert the phoneme in the phoneme sequence. A visual element sequence with a duration is generated by replacing the corresponding visual element. However, here, the phonemes before replacement are also attached to each visual element. This visual element sequence with a continuous length is stored in the visual element sequence storage unit 180.

-Face image vertex clustering-
The clustering processing unit 232 executes the following processing for all combinations of the two face images with respect to the six face images stored in the character model storage unit 156.

  First, the motion vector of the vertex when changing from one face image to the other face image is calculated. One face image is classified into a predetermined number of clusters by performing VQ clustering on the set of motion vectors as described above. As for the movement in the reverse direction, since the direction of the motion vector is only reversed, the clustering is the same in the reverse direction.

  As a result of clustering as described above, the face model after clustering and the representative vertex of each cluster are calculated for all combinations of two face images. This face model is stored in the clustered face model storage unit 234 together with the representative vertex of each cluster.

-Calculation of speech power-
The utterance power calculation unit 238 calculates the average utterance power of each phoneme in the utterance storage unit 152 based on the information on the phonemes attached to each visual element stored in the visual element sequence storage unit 180, and the utterance as the utterance power. The data is stored in the power storage unit 240.

-Creation of animation-
There are various options for creating animations. The first option is a key frame deletion rate γ. This is mandatory because keyframes are always deleted. However, if not specified, a predetermined default value may be used. The second option is to specify whether or not to use the result of clustering in the processing in the key frame deletion unit 236 and the processing in the blend rate adjustment unit 250. The third option is whether or not to perform the processing of the blend rate adjustment unit 244 based on speech power. Furthermore, when executing the processing of the blend rate adjustment unit 244 based on speech power, it is necessary to specify the attenuation rate α. The fourth option is whether or not to perform the process of the blend rate adjustment unit 250. When the processing of the blend rate adjusting unit 250 is performed, it is further necessary to specify the attenuation rate β.

  When it is designated that the processing by the blend rate adjustment unit 244 based on speech power is to be performed, the selection units 242 and 246 give the output of the key frame deletion unit 236 to the blend rate adjustment unit 244 based on speech power. The connection is switched so that the output of the blend ratio adjustment unit 244 is supplied to the selection unit 248. In other cases, the selection units 242 and 246 switch the connection so that the output of the key frame deletion unit 236 is directly given to the selection unit 248.

  On the other hand, when it is designated that the processing by the blend rate adjustment unit 250 is to be performed, the selection units 248 and 252 give the output of the selection unit 246 to the blend rate adjustment unit 250, and visually see the output of the blend rate adjustment unit 250. The connection is switched so as to be given to the elementary sequence storage unit 254. In other cases, the selection units 248 and 252 switch the connection so that the output of the selection unit 246 is directly supplied to the visual element sequence storage unit 254.

  Hereinafter, on the assumption that the processing by the blend rate adjustment unit 244 and the processing by the blend rate adjustment unit 250 by speech power are both selected without losing generality, the case where the model after clustering is not used and the case where the model is used are used. The operations of the key frame deleting unit 236, the speech rate blend rate adjusting unit 244, and the blend rate adjusting unit 250 will be described.

(1) When a model after clustering is not used -Operation of the key frame deletion unit 236-
The key frame deletion unit 236 reads the deletion rate γ input by the deletion rate input unit 201 (FIG. 11, step 360), and deletes it to the number of visual elements in the visual element sequence stored in the visual element sequence storage unit 180. The number K of deleted frames is calculated by multiplying by the rate γ (step 362).

  Further, the key frame deletion unit 236 determines whether or not the key frames corresponding to the number K of deleted frames have been deleted in step 368. Normally, in the first determination, key frames corresponding to the number K of deleted frames are not deleted. Accordingly, the process proceeds to step 370. In step 370, since it is not specified that the model after clustering is used, the process proceeds to step 372.

  In step 372, the distance D between the key frames is calculated using all the vertices between all the adjacent key frames in the visual element sequence. In step 374, the total moving speed of each point is calculated based on this distance. Set the earliest keyframe as the deletion target. In step 380, the key frame is deleted. Thereafter, the process returns to step 366.

  Thereafter, when the number of deleted key frames is larger than the number K of deleted frames, the process is terminated.

  The visual element sequence from which K key frames have been deleted in this way by the key frame deletion unit 236 is provided to the blend rate adjustment unit 244 based on speech power via the selection unit 242.

-Operation of blend rate adjustment unit 244 by speech power-
First, the blend rate adjustment unit 244 based on speech power reads the attenuation rate α (step 450 in FIG. 14). In step 452, based on the information about the phonemes in the visual element sequence output from the key frame deletion unit 236, the average utterance power over the duration of each phoneme is calculated from the utterance data stored in the utterance storage unit 152.

  In step 454, the maximum power MAX (SP) and the minimum power MIN (SP) are calculated from the average utterance power thus calculated, and in step 456, blending is performed for each key frame by an expression using the attenuation factor α. The rate BR (N) is adjusted. The visual element sequence in which the blend rate is adjusted for all the key frames is provided to the blend rate adjusting unit 250 via the selection unit 246 and the selection unit 248.

-Operation of blend rate adjustment unit 250 by vertex speed-
The blend rate adjustment unit 250 based on the vertex velocity first reads the attenuation rate β (FIG. 15, step 470), and multiplies the key frame included in the visual element sequence given from the selection unit 248 by the attenuation rate β. The invariant frame number L is calculated (step 472). In the following step 474, step 476 is selected.

  In step 476, for all key frames in the visual element sequence given from the selection unit 248, the average speed of all vertices between the key frames before and after the key frame is calculated. In step 478, the key frames are sorted in descending order of the average speed using the average speed calculated in this way as a sort key.

  In step 484, the maximum value of the average speed among the L key frames from the lower order of the key frames sorted in step 478 is set to the value of <VS>. In step 486, the blend rate is adjusted for each of the key frames other than the invariant frame by using the value of the speed <VS> set in step 484 and the above-described equation. When the adjustment of the blend rate is completed for all the key frames other than the invariant frames, the visual element sequence for which the adjustment of the blend ratio has been completed is output to the visual element sequence storage unit 254 shown in FIG.

  The blend processing unit 256 reads the visual element sequence stored in the visual element sequence storage unit 254, uses the visual element specified by the key frame as the time corresponding to each key frame, and the frame time between the key frames. Then, an intermediate image is created between the key frames on both sides of the frame by interpolation using the blend rate assigned to the key frame. In this way, an animation is created by creating an image by interpolation using a key frame image and its blend ratio in each frame at a fixed time interval.

(2) When using a model after clustering When using a model after clustering, each unit of the lip sync animation creating apparatus 200 operates as follows.

-Operation of the key frame deletion unit 236-
Referring to FIG. 11, key frame deletion section 236 operates in the same manner as in the case where the model after clustering is not used for the processing from steps 360 to 368. However, step 376 is selected in the determination of step 370. In step 376, the distance D ′ is calculated using the representative vertices between all adjacent key frames. The calculation of the distance D ′ using the representative vertices is as described above, but the movement distance of the representative vertices is multiplied by the number of vertices in the cluster represented by the representative vertices, and the value is applied to all clusters. The distance D ′ is obtained by summing up.

  In step 378, using the distance D 'thus calculated, the key frame with the fastest vertex movement is determined as the key frame to be deleted. The processing after step 380 is the same as when the model after clustering is not used.

-Operation of blend rate adjustment unit 244 by speech power-
The blend rate adjustment unit 244 based on speech power is exactly the same as when the model after clustering is not used. Therefore, details thereof will not be repeated here.

-Operation of Blend Ratio Adjustment Unit 250-
In this case, the blend rate adjustment unit 250 operates as follows. Referring to FIG. 15, the processing in steps 470 and 472 is the same as that in the case where the model after clustering is not used. However, step 480 is selected in the determination of step 474.

  In step 480, for all key frames, the average speed of vertices between the previous and next key frames is calculated using the motion vector of the representative vertex of the cluster to which each vertex belongs. The calculation method here is the same as the calculation method in the key frame deletion unit 236. In step 482, all key frames are sorted in descending order using the average speed thus calculated as a sort key. After this, steps 484 and 486 are executed in the same manner as when the model after clustering is not used.

  FIG. 16 shows an example of the result of key frame deletion by the key frame deletion unit 236. FIG. 16A shows that the key frame deletion unit 236 does not delete the key frame (the output from the visual element sequence generation unit 230 remains as it is, but the blend rate is given an initial value depending on the speech power). 16 (B) and FIG. 16 (C) show the results when the deletion rate γ is set to 20% and 30%, respectively. FIG. 16D shows the result of manually setting a key frame by a human animator while listening to the sound according to a conventional method. It is preferable that results close to FIG. 16D are obtained by automatic processing.

  Comparing FIG. 16A and FIG. 16B, it can be seen that the key frames 500 and 502 are deleted. As a result, FIG. 16B and FIG. 16D are quite close to each other. Further, comparing FIG. 16B and FIG. 16C, the key frame 510 is deleted. When this result is compared with FIG. 16D, it can be seen that both are very similar. In particular, the animation synthesized from the result of FIG. 16C and the animation synthesized from the result of manual work of FIG. 16D are very well matched in the first half, and there is almost no difference in subjective evaluation. There wasn't.

  The upper (A) and (B) of FIG. 17 compare the animation result (A) near the mouth of the face image obtained by the conventional method and the animation result (B) obtained by the above embodiment. Show. The lower sections (C) and (D) of FIG. 17 show the blend ratios of the corresponding key frames. FIG. 17D shows the blend ratio according to the conventional method, and FIG. 17C shows the blend ratio according to the embodiment of the present invention. The face animation of the part corresponding to the frame 530 in FIG. 17C and the frame 532 in FIG. 17D corresponds to FIGS. 17B and 17A.

  Referring to FIGS. 17C and 17D, a blend rate graph 522 according to the conventional method is compared with a blend rate graph 520 according to the present embodiment. As a result, the mouth image moves smoothly.

  As described above, according to the visual element sequence creation unit 230 according to the present embodiment, the voice is automatically supported from the speech voice and its transcription, and the basic face image model corresponding to the visual element. Thus, a face image that changes smoothly can be created. In a portion where the speech power is low or a key frame in which each vertex of the model between the adjacent key frames moves too fast, the blend rate is adjusted to be low. As a result, the animation of the obtained face image is not a so-called “noisy” animation, but an animation that is smooth and close to the case where the key frame and its blend ratio are adjusted manually can be created.

[Realization by computer]
The above-described embodiment can be realized by a computer system and a program executed on the computer system. FIG. 18 shows the appearance of a computer system 550 used in this embodiment, and FIG. 19 is a block diagram of the computer system 550. The computer system 550 shown here is merely an example, and other configurations can be used.

  Referring to FIG. 18, computer system 550 includes a computer 560, a monitor 562, a keyboard 566, a mouse 568, a speaker 558, and a microphone 590, all connected to computer 560. Further, the computer 560 includes a DVD-ROM (Digital Versatile Disk Read-Only-Memory) drive 570 and a semiconductor memory drive 572.

  Referring to FIG. 19, computer 560 further stores a bus 586 connected to DVD-ROM drive 570 and semiconductor memory drive 572, a CPU 576 connected to bus 586, and a boot-up program for computer 560. ROM 578, RAM 580 that provides a work area used by CPU 576 and serves as a storage area for programs executed by CPU 576, and stores voice data, acoustic models, language models, lexicons, and mapping tables It includes a hard disk drive 574 and a network interface 596 that provides a connection to the network 552.

  Speech storage unit 152, transcription storage unit 154, character model storage unit 156, acoustic model storage unit 170, phoneme-visual element mapping table storage unit 176, visual element sequence storage unit 180, clustered face model storage shown in FIG. The unit 234, the speech power storage unit 240, the visual element sequence storage unit 254, and the like are all realized by the hard disk drive 574 or the RAM 580 shown in FIG. Further, the deletion rate input unit 201, the cluster processing specification unit 202, the speech power use instruction input unit 204, the attenuation rate input unit 206, the use instruction input unit 208, the attenuation rate input unit 210, and the like are all shown in FIGS. The monitor 562 shown, and a program that implements a graphical user interface using a keyboard 566 and a mouse 568 are implemented. Since the configuration of such an input program is well known, details thereof will not be described here.

  The reproduction of the facial image animation 260 is realized by an animation reproduction program (not shown). The animation reproduction program itself may be any program that provides a function of sequentially displaying a frame sequence at a fixed frame interval according to a predetermined time table.

  The software for realizing the system of the above-described embodiment is distributed in the form of an object code recorded on a medium such as a DVD-ROM 582 or a semiconductor memory 584, and a reading device such as a DVD-ROM drive 570 or a semiconductor memory drive 572 is installed. To the computer 560 and stored in the hard disk drive 574. When CPU 576 executes a program, the program is read from hard disk drive 574 and stored in RAM 580. An instruction is fetched from an address designated by a program counter (not shown), and the instruction is executed. The CPU 576 reads data to be processed from the hard disk drive 574 and stores the processing result in the hard disk drive 574 as well. The speaker 558 and the microphone 590 are not directly related to the present invention, but the speaker 558 is necessary for the generation of sound during reproduction of the created animation. When the computer system 550 is used for recording speech data, a microphone 590 is required.

  Since the general operation of the computer system 550 is well known, detailed description thereof is omitted.

  Regarding the software distribution method, the software does not necessarily have to be fixed on a storage medium. For example, the software may be distributed from another computer connected to the network. A part of the software may be stored in the hard disk drive 574, and the remaining part of the software may be taken into the hard disk drive 574 via the network and integrated at the time of execution.

  Typically, modern computers utilize the general functions provided by a computer operating system (OS) to achieve functions in a controlled manner according to the desired purpose. Therefore, a control structure that does not include a general function that can be provided from the OS or a third party, and that achieves a desired purpose as a whole even if the program specifies only a combination of execution orders of the general functions. It is obvious that the program is included in the scope of the present invention.

[Second Embodiment]
<Outline>
According to the first embodiment described above, it is possible to create a smooth facial image animation based on voice. However, in an animation as a product, the image is not simply smooth, and various restrictions may be given. For example, a normal animation is created at a frame rate similar to that of a television (30 fps (frame per second)) or a movie (24 fps). However, commercial animation may require that the animation be created at a smaller (slower) frame rate. For example, it may be requested to create an animation at 12 fps, 8 fps, or the like. In such a case, the following problems arise.

  In the apparatus according to the first embodiment, the frame rate at the time of creating an animation is set high, so that a smooth video can be obtained. However, when an animation is created with a low frame rate, a plurality of phonemes are often included in the duration of one key frame. Then, originally, only one type of mouth image is included in a period including a plurality of visual elements. Therefore, it becomes a problem which visual element should be assigned to the mouth image. In this case, it is appropriate to assign one of a plurality of visual elements included in the continuation length of one key frame to the visual element of the key frame. However, in this case, the same visual element may be assigned to successive key frames in some cases. In general, when creating an animation at a slow frame rate of 8 fps, an image having the same frame rate as that of a television, movie, or the like is eventually created. If primes are assigned, the same visual element will continue for a fairly long period of time, and the animation may become unnatural.

  A related problem is that in an animation creation program that is currently used, if a shape is assigned to a key frame and the next key frame, these two images are used for the images of the frames that exist between them. A function of automatically interpolating images of one key frame to create an image of each frame is provided as standard. In such a case, the change of the image between the key frames is smooth without effort, but in the case of an animation created on the assumption of a slow frame rate, a motion different from the intended one is generated. . In the case of a slow frame rate, the resulting animation will move in a “cracking” manner. This is called “limit feeling” and is one technique for creating animation. In an animation intended to generate such a limit feeling, there is a problem that the intended limit feeling cannot be achieved because of such an automatic interpolation function.

  Furthermore, in the case of human speech, it is common to leave the mouth open at the end of the speech, but in animation, it may feel unnatural if the speech is terminated in such a way. Therefore, it can be considered that the mouth is always closed at the end of the utterance. However, in this case, there is a problem of how to correct it so that it looks natural.

  The lip sync animation creating apparatus according to the second embodiment to be described later is for solving these problems.

-Speech termination correction-
First, an animation correction method for correcting the mouth to close at the end of the utterance (hereinafter, this correction will be referred to as “utterance end correction”) will be described. Referring to FIG. 20, it is assumed that a key frame sequence 610 obtained from a speaker's voice includes four consecutive key frames 620, 622, 624, and 626. Of these, the key frame 626 represents a blank period after the utterance.

  In this embodiment, the end position of the key frame 624 corresponding to the end of the utterance is adjusted as follows.

  Referring to FIG. 20, consider an utterance power sequence 630 of a speech signal of a speaker who has created a key frame sequence 610. In the present embodiment, the speech power sequence 630 is traced back on the time axis from the end position of the key frame 624 (start position of the key frame 626), and the speech power is maximized within the period corresponding to the key frame 624. Search for a point 640. Next, the speech power attenuated by a predetermined attenuation amount 642 (δ (dB)) is calculated from the speech power value at this point 640, and the speech power is also attenuated by going back the time axis from the end of the key frame 624. A point 644 that becomes equal to the later speech power is searched. The position on the time axis corresponding to this point 644 is set as the end position of the key frame 624.

  As a result, as shown in FIG. 20, the position of the key frame 626 advances to the position of the point 644 and becomes a new key frame 652, and the continuation length is increased by the reduction of the continuation length of the key frame 624. When an animation is created using the key frame sequence 650 thus obtained, the time when the mouth closes at the end of the utterance is advanced, and the animation becomes natural.

-Frame rate conversion and visual element assignment processing-
Next, a description will be given of a determination method in the present embodiment as to which visual element is assigned to each key frame at a low frame rate. Referring to FIG. 21, it is assumed that a key frame sequence 670 includes six key frames 680, 682, 684, 686, 688 and 690. When the frame rate is slowed down to about 8 fps, the time of the key frame is fixed at the frame position. That is, the key frame and the frame position of the image at the predetermined frame rate match as shown in FIG.

  On the other hand, consider a case where the key frame sequence 670 shown in the upper part of FIG. 21 is generated from the key frame sequence 672 obtained by the lip sync animation apparatus according to the first embodiment. The key frame sequence 672 includes key frames 700, 702, 704, 706, 708, 710, 712, 714, and 716.

  In this case, since the duration of each key frame in the key frame sequence 670 is longer than the duration of each key frame in the key frame sequence 672, the duration of one key frame in the key frame sequence 670 is Visual elements of a plurality of key frames in the key frame sequence 672 correspond to each other. For example, a key frame 682 may be assigned visual elements of three key frames 702, 704, and 706 that are temporally adjacent. Similarly, keyframe 688 may be assigned the visual elements of keyframes 714 and 716. Thus, when there is a possibility that a plurality of visual elements may be assigned to one key frame, it becomes a problem which visual element should be selected.

  By the way, in an actual utterance, it is observed that a mouth movement occurs prior to the generation of voice. Therefore, it is natural that the mouth can be moved to respond to the sound before the sound. In this embodiment, in accordance with such a concept, when visual elements are assigned to each key frame of the key frame sequence 670 shown in FIG. 21, the visual elements are included in the key frame sequence 672 within the duration of the key frame. A visual element of a key frame having the beginning of is assigned.

  For example, with reference to FIG. 22, consider a key frame 682 indicated by an ellipse 730. As described above, the key frame 682 may correspond to the three key frames 702, 704, and 706 in the key frame sequence 672. However, among these, the key frame 702 is not a candidate because the starting end is not within the continuation length of the key frame 682. It is the key frames 704 and 706 that satisfy the condition that the leading edge is within the duration of the key frame 682. When two or more visual elements are present in the key frame 682 as described above, it is natural to assign the visual element generated earlier to the key frame 682. Therefore, in this embodiment, the visual element N (/ m /) of the key frame 704 is assigned to the key frame 682 as indicated by an arrow 734. The visual elements of the two key frames 702 and 706 are not assigned to the key frame 682, as indicated by the dotted arrows 732 and 736.

  In such a case, there is a possibility that a problem occurs in the obtained image. For example, as indicated by an ellipse 740 in FIG. 22, there are key frames 714 and 716 having a start edge within the continuation length of the key frame 688. All of these satisfy the conditions for assigning to the visual element of the key frame 688. However, for example, as shown in FIG. 22, when the visual element A (/ a /) is assigned to the key frame 686 immediately before, the visual element A (/ a / of the key frame 714 is assigned to the key frame 688. ), The visual elements of the two key frames 686 and 688 are exactly the same. As described above, in this case, since the same visual element continues for a considerably long time, there is a problem that the animation becomes unnatural.

  In such a case, the visual element I (/ i /) of the second key frame 716 is assigned to the key frame 688 instead of the key frame 714.

  In this way, a key frame sequence 670 having a considerably low frame rate is created from the key frame sequence 672 originally created assuming a high frame rate, and an artificial face image obtained therefrom is unnatural. You can create something that isn't that long.

  As described above, visual elements indicated by solid arrows 750, 734, 752, 754, and 744 in FIG. 22 are assigned to each key frame of the key frame sequence 670. Note that a visual element of the next key frame not shown in the key frame row 672 is assigned to the key frame 690 shown at the end of the key frame row 670 in FIG.

-Shape stabilization treatment-
By the way, for the limit feeling described above, assume that different visual element mouth shapes are assigned to the key frames 686 and 688 as shown in FIG. In a commonly used animation creation program, an image of a frame between the two key frames is generally generated by performing interpolation between the two key frames. As a result, there is a problem that the intended limit feeling cannot be obtained. This problem will be described with reference to FIG.

  Referring to FIG. 23A, a time corresponding to key frame 686 is set as time t, and a time corresponding to key frame 688 is set as time t + 6. That is, there are five frames between the two key frames. At time t, the blending rate of the visual element / a / in this key frame 790 is 100% as indicated by a circle 770, and the blending rate of the visual element / i / is 0 as indicated by a circle 774. %. On the other hand, at time t + 6, the blending rate of the visual element / i / is 100% as indicated by the circle mark 776, and the blending ratio of the visual element / a / is 0% as indicated by the circle mark 772. Become. And the blend ratio of both is calculated as shown by blend ratio curves 780 and 782. In each frame between time t and time t + 6, a face image is created by blending the face images of the two visual elements according to the blend ratio. When such blending is performed, the image changes smoothly, but there is a problem in that the sense of limit is lost, and the request to create an animation at a small frame rate cannot be satisfied.

  Therefore, in this embodiment, as shown in FIG. 23B, at the time t + 5 corresponding to the frame immediately before the time t + 6, the blend ratio of the visual elements / a / and / i / at the time t is left as it is. The key frame 790 is copied as the key frame 792. As a result, even when automatic blending is performed by the animation creating program, the blend ratio of visual element / a / is 100% as shown by straight lines 800 and 802 between times t and t + 5. The blend ratio of elementary / i / is maintained at 0%. The change of the face image is performed between times t + 5 and t + 6, and the above-described limit feeling can be achieved.

<Configuration>
FIG. 24 shows a block diagram of a lip sync animation creating apparatus 810 according to the second embodiment. The configuration of the lip sync animation creating apparatus 810 is substantially the same as that of the lip sync animation creating apparatus 200 according to the first embodiment shown in FIG. 5, but the key frame deleting unit 236 and the selecting unit shown in FIG. 242 further includes an utterance end correction unit 822 for correcting the utterance end, and selection units 820 and 824 for selecting whether or not to use the function of the utterance end correction unit 822. And a frame rate conversion for converting the frame rate of the visual element sequence with duration to the frame rate specified by the frame rate input 832. Unit 840 and the visual element sequence output by frame rate conversion unit 840 A shape stabilization processing unit 842 for executing a shape stabilization process for preventing a sense of limit from being lost by blending by a program, and a visual element with a continuous length after frame rate conversion output from the shape stabilization processing unit 842 A visual element sequence storage unit 846 with a continuous length for storing a sequence, and first and second inputs connected to outputs of visual element sequence storage units 254 and 846 with a continuous length, respectively, for frame rate conversion In accordance with the instruction of the use instruction input 830 for designating whether or not to use, either the output of the visual element sequence storage unit with duration 254 or the output of the visual element sequence storage unit with duration 884 is selected and the blend processing unit The lip-sync animation creating apparatus 2 shown in FIG. 0 and is different.

  Note that the selection units 820 and 824 shown in FIG. 24 output the output of the key frame deletion unit 236 via the utterance end correction unit 822 according to the use instruction input 826 that specifies whether or not to perform utterance end correction. The processing given to 242 and the processing given directly to the selection unit 242 without going through the utterance end correction unit 822 are selectively performed. Further, the input 828 of the attenuation rate δ (dB) described with reference to FIG. 20 is given to the speech end correction unit 822. The use instruction input 826 and the use instruction input 830 may use the same instruction.

  As described above, the utterance end correction unit 822, the frame rate conversion unit 840, and the shape stabilization processing unit 842 of the lip-sync animation creating apparatus 810 include computer hardware and a computer program executed on the hardware. And can be realized. Hereinafter, the control structure of these programs will be described.

  FIG. 25 is a flowchart showing a control structure of a computer program for realizing the utterance end correction unit 822.

  Referring to FIG. 25, the program determines whether there is an unprocessed utterance end in step 870 for searching for an unprocessed utterance end in the key frame sequence output from the key frame deleting unit 236, If there is no utterance termination, the process ends. If there is an utterance termination, the determination step 872 moves to the next step, and if it is determined in the determination step 872 that there is an unprocessed utterance termination, And determining a maximum value Pmax of the audio power within the visual element duration of the key frame immediately before the end of the speech.

  The process of searching for an unprocessed utterance end in step 870 is performed by searching for a key frame to which a visual element other than a blank is assigned immediately before a key frame to which a blank visual element is assigned. The processing for obtaining the maximum value Pmax performed in step 874 is as described with reference to FIG. The point giving the maximum value Pmax here corresponds to the point 640 in FIG.

  The program further satisfies such a condition, step 876, after step 874, going back from the end of the visual element duration being processed and determining the first time t when the speech power is Pmax-δ (dB). It is determined whether or not there is a point. If there is no point that satisfies the condition, the process branches to step 870. If there is a point that satisfies the condition, step 878 is branched to the next step. Is executed in response to the determination that there is a point satisfying the condition, and the end of the visual element continuation length is changed to the time t found in step 876, and at the same time, the start of the immediately following key frame And step 880 for performing a process of changing to a time t. After step 880, control returns to step 870. The point of time t obtained in step 876 corresponds to the point 644 described in FIG.

  FIG. 26 is a flowchart showing a control structure of a computer program for realizing the frame rate conversion unit 840 shown in FIG. Referring to FIG. 26, this program performs step 900 for setting a value 0 to variable i representing the number of key frames to be processed in the subsequent iterative processing, step 902 for adding 1 to variable i, and step 902. As a result of the addition process, it is determined whether or not the variable i is larger than the number of all key frames. If the variable i becomes larger, the process is terminated. Otherwise, the control branches to the next step. Step 904.

  The program is further executed in response to determining in step 904 that the variable i is not greater than the number of key frames and writing the i-th key frame (hereinafter this key frame is referred to as “key frame (i)”. )) Searching for a visual element whose starting edge is included in the continuation length of step 906, determining whether or not the number N of visual elements found in step 906 is 0, and branching the process depending on the result 908; Step 908 is executed in response to the determination that the number of visual primes N = 0, discards the key frame (i), and returns control to step 902. This is executed in response to the determination that the number N of primes is not 0, determines whether the number N of visual primes is 1, and branches control according to the determination result Step 912 is executed, and in response to the determination that the number N of visual elements is determined to be 1 in Step 912, the visual element found in Step 906 (this visual element) is displayed in the key frame (i). Since there is only one prime in this case, this is written as visual prime (1).) And control is returned to step 902. In step 912, it is determined that the number N of visual primes is not 1. And a step 916 of setting 0 to a variable j representing the number from the head of a visual element that is executed in response and includes the start edge in the duration of the key frame (i) in the subsequent processing.

  The program further includes step 918 of adding 1 to variable j following step 916, and the result of addition in step 918 is that the value of variable j is greater than the number N of visual elements in key frame (i). A step 920 for branching the control according to the determination result, and a key frame executed in response to the determination that the value of the variable j is greater than the number N of visual elements in step 920. In step 922, the head visual element (visual element (1)) having the start end in key frame (i) is assigned to (i), and control returns to step 902. In step 920, the value of variable j is the visual element. It is executed in response to the determination that it is not greater than the number N, and the j-th visual element in the key frame (i) (this is referred to as “visual element (j)”). Ki Determines whether identical or not and visual elements of the frame (key frame (i-1)), and a step 924 for branching the control according to the determination result.

  If it is determined in step 924 that the visual element (j) matches the visual element of the key frame (i-1), control returns to step 918, otherwise control proceeds to the next.

  The program is further executed in response to determining in step 924 that the visual element (j) is not the visual element of the key frame (i-1), and assigning the visual element (j) to the key frame (i). Step 926 includes processing for assigning and returning control to Step 902.

  FIG. 27 shows a control structure of a program for realizing the shape stabilization processing unit 842 shown in FIG. 24 in a flowchart format. Referring to FIG. 27, this program executes step 950 for setting 1 to a variable i representing the number of a key frame to be processed in the subsequent processing, step 952 for adding 1 to variable i, and step 952. As a result of the addition processing of step 954, it is determined whether or not the value of the variable i has become larger than the number of key frames to be processed. If the value of the variable i exceeds the number of key frames, step 954 and step 954 are ended. Is executed in response to the determination that the value of the variable i does not exceed the number of key frames, and the key frame (i-1) is copied to the frame immediately before the key frame (i) to create a new one. Step 956 and the like are included for performing processing to make a key frame and then returning control to Step 952.

<Operation>
The lip sync animation creating apparatus 810 shown in FIG. 24 operates as follows. In the following description, it is assumed that the usage instruction inputs 826 and 830 indicate the same value to the lip sync animation creating apparatus 810. When the usage instruction inputs 826 and 830 are values specifying that the processing by the speech end correction unit 822, the processing by the frame rate conversion unit 840, and the processing by the shape stabilization processing unit 842 are not used, the selection units 820 and 824 are used. Gives the output of the key frame deletion unit 236 directly to the input of the selection unit 242. The selection unit 848 gives the output of the visual element sequence storage unit 254 with a continuous length to the blend processing unit 256. Therefore, in this case, the configuration of the lip sync animation creating apparatus 810 is substantially the same as that of the lip sync animation creating apparatus 200 shown in FIG. 5, and the same operation as the lip sync animation creating apparatus 200 is performed.

  When the usage instruction inputs 826 and 830 are values specifying that the speech termination correction unit 822, the frame rate conversion unit 840, and the shape stabilization processing unit 842 are used, the selection unit 820 outputs the key frame deletion unit 236. Is provided to the utterance end correction unit 822. The output of the utterance end correction unit 822 is given to the input of the selection unit 242 via the selection unit 824.

  On the other hand, the selection unit 848 selects not the output of the visual element sequence storage unit with duration 254 but the output of the visual element sequence storage unit with duration 884, and supplies the selected result to the blend processing unit 256. In response to the frame rate input 832, the frame rate conversion unit 840 sequentially reads the visual element sequences stored in the visual element sequence storage unit 254 with a continuous length, and calculates the frame rate using the method shown in FIGS. 21 and 22. Then, the visual element is assigned to each frame, and the visual element sequence after the frame rate conversion is given to the shape stabilization processing unit 842. The shape stabilization processing unit 842 performs a process of copying each key frame to the frame immediately before the next key frame in the visual element sequence output from the frame rate conversion unit 840. This process is as shown in FIG. After performing this process for all the key frames, the result is output to the visual element sequence storage unit 846 with a continuous length.

  As described above, the selection unit 848 selects the output of the visual element sequence storage unit 846 with a continuous length and supplies it to the blend processing unit 256. The blend processing unit 256 reads the key frame sequence stored in the visual element sequence storage unit with length of continuation 846, and interpolates the specified blend ratio between adjacent key frames in the frame between them. Create and output an animation. The frame rate of the animation 260 created in this way is the same as the frame rate of the television or movie, but the key frame is deleted by the frame rate conversion unit 840 and further the adjacent key frame is deleted by the shape stabilization processing unit 842. Since the shape stabilization process is performed to prevent interpolating the animation between the frames, the limit is substantially the same as the low frame rate animation according to the frame rate value specified by the frame rate input 832 A feeling can be obtained.

[Third Embodiment]
<Outline>
According to the first and second embodiments described above, visual elements / A /, / I /, / U /, / E /, / O /, and / N / (hereinafter referred to as “standard visual elements”, these The phoneme corresponding to is called a “standard phoneme”). However, in the case of Japanese, there are more than ten types of visual elements including standard visual elements (/ K /, / S /, / T /, etc.), so a smooth animation in Japanese is created with only standard visual elements. May not be enough. In the above embodiment, the face image for the standard visual element has been prepared in advance. However, if a Japanese animation is to be created using another visual element, the face image to be prepared must be prepared. The number increases. Because the face model for these face images is created by manually editing the standard face model that is used as a reference for animation creation, it is difficult to prepare face images for many visual elements. It is. When creating a foreign language animation such as English, Chinese, etc., face images must be created for different visual elements, and therefore more difficult.

  A lip-sync animation creating apparatus according to a third embodiment described below includes a visual element including a standard visual element and a visual element other than the standard visual element (hereinafter referred to as “general visual element”). This is for creating Japanese lip-sync animations using groups and extending them to multiple languages.

<Configuration>
FIG. 28 shows a block diagram of a lip sync animation creating apparatus 1000 according to the third embodiment. The configuration of the lip sync animation creating apparatus 1000 shown in FIG. 28 is almost the same as the configuration of the lip sync animation creating apparatus 810 according to the second embodiment shown in FIG. This is different from the lip sync animation creating apparatus 810 shown in FIG. 24 in that it is for creating a Japanese facial image animation 260 using visual elements.

  Specifically, the lip sync animation creating apparatus 1000 has the same configuration as that of the phoneme-visual element mapping table storage unit 176 shown in FIG. 24, but for each Japanese phoneme, For storing a phoneme-visual element mapping table different from the phoneme-visual element mapping table storage unit 176 shown in FIG. 24 in that one visual element is associated from the visual element group including other visual elements. Instead of the point including the phoneme-visual element mapping table storage unit 1002 and the 3D character model storage unit 156 storing the face model corresponding to the standard visual element (hereinafter referred to as “standard visual element model”) shown in FIG. A face model based on the standard face model for not only standard visual elements but also other Japanese visual elements (hereinafter referred to as “general visual element model”). Called.) Is different from the lip-sync animation creating apparatus 810 shown in FIG. 24 at a point that includes a 3D character model storage unit 1004 for storing a 3D character model consisting of.

  The lip sync animation creating apparatus 1000 further generates three-dimensional data (hereinafter referred to as “capture data”) of facial feature points captured when a certain speaker is speaking a Japanese sentence. Capture data storage unit 1006 for storing the phoneme in association with the phoneme, standard visual element model storage unit 1008 for storing the standard visual element model, and capture data and standard visual element model storage unit stored in the capture data storage unit 1006 Using the standard visual element model stored in 1008, each of the capture data corresponding to phonemes other than the standard phonemes (/ k /, / s /, / t /, etc.) is converted into the capture data corresponding to the standard phonemes. A coefficient calculation unit 1010 for calculating a coefficient for approximation by linear sum, and a coefficient calculated by the coefficient calculation unit 1010 The general visual element model is represented by a linear sum of the standard visual element models stored in the standard visual element model storage unit 1008, and a 3D character model is created by using the standard visual element model and the general visual element model. It differs from the lip sync animation creating apparatus 810 shown in FIG. 24 in that it includes a character model combining unit 1012 for storing in the model storage unit 1004.

  Design how many general visual elements, how to select as general visual elements, and which visual element of Japanese visual elements to associate with each Japanese phoneme Belongs to matter. However, it is necessary to always associate standard phonemes with standard visual elements.

  Referring to FIG. 29, using the capture data stored in capture data storage unit 1006 in FIG. 28 and the standard visual element model stored in standard visual element model storage unit 1008, a linear sum by the standard visual element model is used. A process for obtaining a coefficient for approximating the general visual element model will be described.

  Referring to FIG. 29, the capture data storage unit 1006 stores Japanese phonemes / a /, / i /, / u /, / e /, / o /, / n /, / k /, / s /, Capture data of the speaker's face when speaking / t /, / h /, / b /, etc.

等がそれぞれ記憶されているものとする。／〜Ｎ／（記号「〜」は式中文字の上に付されている。）は、音素／ｎ／を発話中の発話者の顔の特徴点のキャプチャデータである。／〜Ｎ／以外のキャプチャデータはいずれも、／〜Ｎ／を基準とし、顔画像の各特徴点が、顔画像の定義されている３次元空間において、キャプチャデータ／〜Ｎ／の対応する特徴点からどの程度移動しているかを示す３次元ベクトル情報によって表されたものである。

Etc. are stored. / ˜N / (symbol “˜” is added above the character in the formula) is capture data of the feature point of the face of the speaker who is speaking the phoneme / n /. Capture data other than / ˜N / is based on / ˜N /, and each feature point of the face image corresponds to the feature corresponding to the capture data / ˜N / in the three-dimensional space where the face image is defined. It is represented by three-dimensional vector information indicating how much the point has moved.

  Referring to FIG. 29, standard visual element model storage unit 1008 uses standard visual element models / A /, / I /, / U /, / E /, and / O / as reference visual element models. Stored in the form of a set of movement vectors of each feature point from / N /. Each of these visual element models is created for a standard visual element model used as an animation character.

  The function of the coefficient calculation unit 1010 is as follows. Here, as an example, a method for obtaining the general visual element model / K / for creating an animation associated with the phoneme / k / from the capture data stored in the capture data storage unit 1006 will be described.

  The general visual element model / ˜K / is formulated as follows.

ただし、〜α_ＫＡ、〜α_ＫＩ、〜α_ＫＵ、〜α_ＫＥ、及び〜α_ＫＯ（記号「〜」は式中文字の上に付されている。）は実数の値をとる変数であり、ε_Ｋは誤差変数である。この式は、一般視覚素モデル／〜Ｋ／を構成する各特徴点の位置を表すベクトルの全てについてたてることができる。すなわち、キャプチャデータを構成する特徴点の数がＭ個であれば、Ｍ個のベクトルの線形和の等式が得られる。

However, ~ α _KA , ~ α _KI , ~ α _KU , ~ α _KE , and ~ α _KO (symbol "~" is attached on the letter in the formula) is a variable that takes a real value, ε _K is an error variable. This equation can be established for all of the vectors representing the positions of the feature points constituting the general visual element model / ˜K /. That is, if the number of feature points constituting the capture data is M, an equation of linear sum of M vectors can be obtained.

Calculate ~ α _KA , ~ α _KI , ~ α _KU , ~ α _KE , and ~ α _KO such that the sum of squares of ε _K calculated for all of these M vector linear sum equations is minimized. To do. The calculated values of ˜α _KA , ˜α _KI , ˜α _KU , ˜α _KE , and ˜α _KO are taken as α _KA , α _KI , α _KU , α _KE , and α _KO , respectively. The processing performed by the coefficient calculation unit 1010 is to calculate this coefficient.

The function of the character model synthesizing unit 1012 uses each of the coefficients α _KA , α _KI , α _KU , α _KE , and α _KO calculated by the coefficient calculating unit 1010 to use each of the feature points constituting the general visual element model. The value of the three-dimensional vector representing the position is calculated as a linear sum of the standard visual element model and stored in the character model storage unit 1004.

  Hereinafter, the function of the character model combining unit 1012 will be described by taking as an example the case of calculating a general visual element model / K / for creating an animation that is associated with a phoneme / k /. The character model composition unit 1012 calculates the general visual element model / K / according to the following equation.

この式は、一般視覚素モデル／Ｋ／を構成する特徴点の位置を表す３次元ベクトルの全てを、標準視覚素モデル／Ａ／、／Ｉ／、／Ｕ／、／Ｅ／及び／Ｏ／を構成する特徴点の位置を表す３次元ベクトルの線形和で表すことを意味する。

This equation represents all of the three-dimensional vectors representing the positions of feature points constituting the general visual element model / K / with the standard visual element models / A /, / I /, / U /, / E / and / O /. Is represented by a linear sum of three-dimensional vectors representing the positions of feature points.

  Similarly, the character model synthesis unit 1012 converts the general visual element models / S /, / T /, / H /, and / B / etc. into the standard visual element models / A /, / I /, / U /, Obtained as a linear sum of / E / and / O /.

  The general visual element model thus obtained is stored in the character model storage unit 1004 together with the standard visual element model.

  Table 7 shows an example of the mapping table stored in the phoneme-visual element mapping table storage unit 1002.

テーブル７を参照して、本実施の形態では、上から１行目の音素／ａ／から５行目の／ｏ／までは、第１の実施の形態で用いられたテーブル１と同様である。ただし、テーブル１と異なり、音素／ｎ／は視覚素／Ｎ／にのみ対応付けられている。７行目では、音素／ｋ／が、一般視覚素／Ｋ／に対応付けられている。８行目以下の音素／ｓ／等についても７行目の音素／ｋ／と同様である。このようなマッピングテーブルを用いると、音素が与えられるとそれに対応する視覚素が分かり、その視覚素のラベルと一致する視覚素ラベルを持つ視覚素モデルをキャラクタモデル記憶部１００４から読出すことができる。

Referring to Table 7, in the present embodiment, the phonemes / a / in the first row from the top to / o / in the fifth row are the same as those in Table 1 used in the first embodiment. . However, unlike Table 1, phonemes / n / are associated only with visual elements / N /. In the seventh line, phoneme / k / is associated with general visual element / K /. The phonemes / s / etc. in the 8th row and below are the same as the phonemes / k / in the 7th row. By using such a mapping table, when a phoneme is given, a visual element corresponding to the phoneme is known, and a visual element model having a visual element label that matches the label of the visual element can be read from the character model storage unit 1004. .

<Operation>
The lip sync animation creating apparatus 1000 whose configuration has been described above operates as follows. The operation of the lip sync animation creating apparatus 1000 shown in FIG. 28 is almost the same as that of the lip sync animation creating apparatus 810 shown in FIG. 24, and only the Japanese 3D character model used is different. Therefore, in the following, only the operation of the lip sync animation creating apparatus 1000 when creating the 3D character model including the general visual element model added in the present embodiment will be described in detail, and the explanation of the other operations will be described. The outline will not be repeated.

  In the lip-sync animation creating apparatus 1000 according to the present embodiment, in order to create the face image animation 260, preparation work of creating a phoneme-visual element mapping table and creating a 3D character model including a general visual element model is performed. is required. The preparatory work is described below.

-Creation of phoneme-visual element mapping table 1002-
A Japanese phoneme and a visual element are manually associated with each other to create a machine-readable phoneme-visual element mapping table and store it in the phoneme-visual element mapping table storage unit 1002. At this time, unlike the second embodiment, phonemes other than standard phonemes do not have to be associated with standard visual elements. An arbitrary phoneme may be associated with a visual element other than the standard visual element (general visual element). An example of the phoneme-visual element mapping table created in this way is the table 7 described above.

-Creation of Japanese 3D character model storage unit 1004-
The coefficient calculation unit 1010 and the character model synthesis unit 1012 create a 3D character model including the general visual element model as well as the standard visual element model as follows. The general visual element models to be created here are all visual elements associated with phonemes in the above-described phoneme-visual element mapping table.

  Referring to FIG. 29, coefficient calculation unit 1010 selects an arbitrary phoneme-visual element pair associated with a phoneme in the phoneme-visual element mapping table, and capture data stored in capture data storage unit 1006 Among the data, capture data (referred to as “compositing target capture data” for convenience) labeled with the phoneme label of the selected pair is read. The coefficient calculation unit 1010 further reads out all the capture data corresponding to the standard phonemes among the capture data stored in the capture data storage unit 1006. Then, as already described, a coefficient for approximating the synthesis target capture data with a linear sum of the capture data corresponding to the standard phonemes is calculated. Then, a visual element label associated with the phoneme of the synthesis target capture data is attached to the coefficient group and given to the character model synthesis unit 1012.

  The coefficient calculation unit 1010 repeats the same processing for all phoneme-visual element mappings stored in the phoneme-visual element mapping table storage unit 1002 including general visual elements.

  The character model synthesis unit 1012 performs the following processing based on the coefficient group and the visual element label given from the coefficient calculation unit 1010. That is, the character model synthesis unit 1012 represents the general visual element model corresponding to the given visual element label as a linear sum of the standard visual elements stored in the standard visual element model storage unit 1008, and at this time, the coefficient The coefficient given from the coefficient calculation unit 1010 is used. As a result, the general visual element model corresponding to the given visual element label is expressed as a linear sum of the standard visual element models.

  The character model synthesizing unit 1012 repeats the above-described processing for all the sets including the coefficient group and the visual element label given from the coefficient calculating unit 1010 and causes the character model storage unit 1004 to store the result. A corresponding visual element label is attached to the general visual element model stored in the character model storage unit 1004.

  The character model composition unit 1012 also stores the standard visual element model stored in the standard visual element model storage unit 1008 in the character model storage unit 1004 with a corresponding visual element label.

  With the above processing, a 3D character model for Japanese is completed.

  When the 3D character model is completed, the subsequent operation of the lip sync animation creating apparatus 1000 is not different from that of the lip sync animation creating apparatus 810 according to the second embodiment. However, as a face image used for an animation key frame, not only a standard visual element model but also a general visual element model can be used. For this reason, the created lip sync animation is smoother than that obtained in the second embodiment.

[Extension to multiple languages]
In the description of the third embodiment described above, it is assumed that the lip sync animation creating apparatus 1000 is an apparatus for creating a Japanese animation. However, the method for creating a Japanese 3D character model in the third embodiment is actually a Japanese standard phoneme and standard visual element model for creating animation in a language different from Japanese, such as English and Chinese. Can be extended using As long as such a 3D character model is used, the basic part of the configuration of the part for creating the lip sync animation in the lip sync animation creating apparatus 1000 can be used as it is.

  For example, the concept for creating an English animation will be described. Since the language used is English, the following changes are required in the lip sync animation creating apparatus 1000 shown in FIG. Assuming that the speakers are different, it is necessary to change the acoustic model stored in the acoustic model storage unit 170 to one corresponding to an English speaker. Naturally, the utterance storage unit 152 and the transcription storage unit 154 for creating an animation also change. The phoneme-visual element mapping table storage unit 1002 also needs to be newly created based on the English phoneme and the visual element when the phoneme is pronounced. Since it is assumed that the speakers are different, the capture data stored in the capture data storage unit 1006 must also be recorded from an English speaker.

  In this case, the 3D character model stored in the character model storage unit 1004 is created as follows. FIG. 30 illustrates a method for preparing a 3D character model for creating an English animation.

  Referring to FIG. 30, in this case, the capture data storage unit 1006 shown in FIG. 29 prepares capture data representing the position of the feature point of the speaker's face when speaking English. This capture data is represented by how much each feature point has moved from the silent position with reference to the silent capture data after removing global coordinate fluctuations due to head swing by correction. The capture data includes capture data at the time of utterance of phonemes corresponding to Japanese standard phonemes.

  The coefficient calculation unit 1010 refers to the English phoneme-visual element mapping stored in the phoneme-visual element mapping table storage unit 1002, and for each phoneme-visual element combination that appears there, The coefficient group is determined on the basis of the least square so that the labeled capture data is approximated by a linear sum of the capture data when a phoneme corresponding to a Japanese standard phoneme is uttered. For all phonemes appearing in the phoneme-visual element mapping table, a general visual element model is created by a linear sum using this coefficient group, stored in the character model storage unit 1004 together with the standard visual element model, and corresponding visual element labels Is attached.

  As described above, an English phoneme-visual element mapping table is prepared, an English 3D character model is prepared, an English speaker acoustic model storage unit 170 is prepared, an English speech storage unit 152 When the transcription storage unit 154 is prepared, an English lip sync animation can be created in the same manner as when a Japanese lip sync animation is created in the third embodiment. The general visual elements stored in the character model storage unit 1004 are all expressed as a linear sum of Japanese standard visual elements, but the linear sum is obtained based on English capture data. , Can reproduce well the face image when speaking in English.

  The above description relates to the case of creating an English lip sync animation using a standard Japanese face model. However, as is apparent from the above description, the lip sync animation creating apparatus 1000 according to the third embodiment is not limitedly applicable only to such language combinations. If capture data representing the three-dimensional position of the face image of the speaker at the time of utterance can be obtained for any combination of languages, the lip sync animation is applied in the same manner to apply the lip sync animation. Can be created.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim in the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

It is a figure which shows the outline of the animation preparation process 30 by the animation preparation apparatus concerning the 1st Embodiment of this invention. It is a figure which shows the face image corresponding to the visual element used in the 1st Embodiment of this invention. It is a figure for demonstrating the concept of a blend rate. It is a figure which shows the example of a change of the face image by blending. It is a block diagram which shows the schematic functional structure of the lip-sync animation production apparatus 200 which concerns on the 1st Embodiment of this invention. FIG. 6 is a more detailed block diagram of the visual element sequence creation unit 230 of FIG. 5. It is a figure which shows the image around a mouth among the visual elements corresponding to each phoneme. It is a figure for demonstrating the motion vector between two visual elements. It is a figure which shows the example of the face image after clustering. It is a figure which shows the other example of the face image after clustering. It is a flowchart which shows the control structure of the computer program which implement | achieves the key frame deletion part 236 of FIG. It is a figure for demonstrating deletion of a key frame. It is a figure for demonstrating the calculation method of average speech power. It is a flowchart which shows the control structure of the computer program which implement | achieves the blend rate adjustment part 244 by the speech power of FIG. It is a flowchart which shows the control structure of the computer program which implement | achieves the blend rate adjustment part 250 by the vertex speed of FIG. It is a figure which compares and shows the production | generation result of the key frame on various conditions in embodiment of this invention, and the designation | designated result of the key frame by manual labor. It is a figure which shows the result of the animation obtained by one embodiment of this invention compared with the thing by the conventional method. FIG. 17 is a diagram illustrating a hardware appearance of a computer system 550. 2 is a block diagram of a computer system 550. FIG. It is a schematic diagram for demonstrating the outline of speech termination | terminus correction | amendment in the 2nd Embodiment of this invention. It is a schematic diagram which shows the concept of frame rate conversion in the 2nd Embodiment of this invention. It is a schematic diagram for demonstrating the determination method of the visual element allocated with respect to each key frame after frame rate conversion in 2nd Embodiment. It is a schematic diagram for demonstrating the shape stabilization process in 2nd Embodiment. It is a schematic block diagram of the lip-sync animation production apparatus 810 which concerns on 2nd Embodiment. It is a flowchart of the computer program for implement | achieving the speech end correction | amendment part 822 shown in FIG. It is a flowchart of the computer program for implement | achieving the frame rate conversion part 840 shown in FIG. It is a flowchart of the computer program for implement | achieving the shape stabilization process part 842 shown in FIG. It is a schematic block diagram of the lip-sync animation production apparatus 1000 which concerns on 3rd Embodiment. It is a more detailed figure for preparing the 3D character model memorize | stored in the character model memory | storage part 1004 of FIG. It is a detailed diagram for creating an English animation.

Explanation of symbols

40 Speaker 42 Audio signal 44 Script 50-58 Phoneme 60-68, 80 Facial image 152 Speech storage unit 154 Transcription storage unit 156, 1004 Character model storage unit 170 Acoustic model storage unit 172 Phoneme segmentation unit 174 Phoneme sequence storage unit 176,1002 Phoneme-visual element mapping table storage unit 178 Phoneme-visual element conversion processing unit 180,254 Visual element sequence storage unit 182 Animation creation unit 200,810,1000 Lip sync animation creation device 202 Cluster processing designation unit 204 Use of speech power Instruction input unit 230 Visual element sequence creation unit 232 Clustering processing unit 234 Clustered face model storage unit 236 Key frame deletion unit 238 Speech power calculation unit 240 Speech power storage unit 44 Blend rate adjusting unit based on speech power 250 Blend rate adjusting unit based on vertex speed 256 Blend processing unit 260 Animation of face image 610, 650, 670, 672 Key frame sequence 620, 622, 624, 626, 680, 682, 684, 686 , 688, 690, 700, 702, 704, 706, 708, 710, 712, 714, 716, 790, 792 Key frame 822 Speech termination correction unit 840 Frame rate conversion unit 842 Shape stabilization processing unit 1006 Capture data storage unit 1008 Standard visual element model storage unit 1010 Coefficient calculation unit 1012 Character model synthesis unit

Claims (21)

Input using a statistical acoustic model prepared in advance, a mapping definition between phonemes and visual elements prepared in advance, and face models of a plurality of facial images prepared in advance corresponding to the visual elements. A lip sync animation creation device for creating a lip sync animation from utterance data,
Using the statistical acoustic model, the mapping definition, and the transcription for the utterance data, a phoneme and a corresponding visual element included in the utterance data are obtained, and a duration with a default blend rate is given. A visual element sequence generating means for generating a visual element sequence, wherein a key frame is defined at a predetermined position within the duration of the visual element sequence, and is defined within the duration of each visual element of the visual element sequence Keyframe sequence is defined by the keyframe
The lip-sync animation creating device further includes, in order from the highest speed of change between the key frames in the key frame sequence and the face model corresponding to the visual element, between the adjacent key frames. And a key frame deleting means for deleting a predetermined percentage of key frames,
A lip sync animation creating apparatus, comprising: blend processing means for creating an animation of a face image by blending between key frames based on a key frame sequence in which some key frames are deleted by the key frame deleting means. The key frame deletion means is configured to select, from among key frames in the key frame sequence, feature points constituting a face model corresponding to a visual element of the key frame and a face model corresponding to a visual element of an adjacent key frame. The lip sync animation creating apparatus according to claim 1, further comprising means for deleting a predetermined percentage of key frames in order from the largest change speed between corresponding corresponding feature points. . Motion vector calculating means for calculating a motion vector between feature points constituting the face model for all combinations of two face models selected from the plurality of face models;
The feature points of the two face models are clustered by a predetermined clustering method for the motion vector calculated by the motion vector calculating means, and a representative vector of each cluster is calculated to create a clustered face model Means for
Clustered face model storage means for storing the clustered face model;
The key frame deletion means includes
For each key frame in the key frame sequence, a clustered face model corresponding to a combination of a visual element of the key frame and a visual element of an adjacent key frame is read from the clustered face model storage means. A moving amount calculating means for calculating the speed of change between key frames of feature points belonging to each cluster using a representative vector of the cluster;
2. The lip sync according to claim 1, further comprising: means for deleting a predetermined percentage of key frames from the key frame sequence in order from the highest speed of change calculated by the movement amount calculating means. Animation creation device. The utterance power for receiving the key frame sequence in which some key frames are deleted by the key frame deletion means and calculating the utterance power of the phoneme corresponding to the visual element of the key frame in the key frame sequence from the utterance data A calculation means;
For each key frame in the key frame sequence, a predetermined value is set such that the smaller the average utterance power calculated by the utterance power calculation means for the duration of the visual element including the key frame, the smaller the blend rate. And a blend rate adjusting means by utterance power for adjusting the blend rate by a function,
2. The lip sync animation according to claim 1, wherein the blend processing unit creates an animation of a face image by blending between key frames based on a key frame sequence in which the blend rate is adjusted by the blend rate adjusting unit based on the speech power. Creation device. Receiving a key frame sequence in which some of the key frames have been deleted by the key frame deleting means, and forming a face model corresponding to the visual element of the key frame and a face model corresponding to the visual element of the adjacent key frame A speed of change calculation means for calculating the speed of change between the vertices constituting
Of the key frames included in the key frame sequence in which some key frames have been deleted by the key frame deletion means, the change speed calculated by the change speed calculation means is higher than a predetermined threshold value. A blend rate adjusting means by vertex speed for updating the blend rate using a predetermined function such that the blend rate becomes a smaller value for a large key frame,
2. The lip sync animation according to claim 1, wherein the blend processing unit creates an animation of a face image by blending between key frames based on a key frame sequence whose blend rate is adjusted by the blend rate adjusting unit based on the vertex speed. Creation device. Motion vector calculating means for calculating a motion vector between feature points constituting the face model for all combinations of two face models selected from the plurality of face models;
The feature points of the two face models are clustered by a predetermined clustering method for the motion vector calculated by the motion vector calculating means, and a representative vector of each cluster is calculated to create a clustered face model Means for
Clustered face model storage means for storing the clustered face model;
The lip sync animation creating device further includes:
Clustering corresponding to a combination of a visual element of the key frame and a visual element of an adjacent key frame, out of each key frame, upon receiving a key frame sequence in which some key frames are deleted by the key frame deleting unit The calculation of the change speed for reading the combination of the face models from the clustered face model storage means and calculating the change speed between the key frames of the feature points belonging to each cluster using the representative vector of the cluster Means,
Of the key frames included in the key frame sequence in which some key frames have been deleted by the key frame deletion means, the change speed calculated by the change speed calculation means is higher than a predetermined threshold value. A blend rate adjusting means by vertex speed for updating the blend rate using a predetermined function such that the blend rate becomes a smaller value for a large key frame,
2. The lip sync animation according to claim 1, wherein the blend processing unit creates an animation of a face image by blending between key frames based on a key frame sequence whose blend rate is adjusted by the blend rate adjusting unit based on the vertex speed. Creation device. Lip sync animation from input speech data using statistical acoustic models prepared in advance, mapping definitions between phonemes and visual elements prepared in advance, and face models of multiple facial images prepared in advance A lip-sync animation creation device for creating a speech, the transcription for the utterance data is available,
Using the statistical acoustic model, the mapping definition, and the transcription, the phoneme sequence and the corresponding visual element included in the utterance data are obtained, and a visual element sequence with a duration is given a default blend rate. Including a visual element sequence creation means for creating
A key frame is defined at a predetermined position within the duration of the visual element sequence, and a key frame sequence is defined by a key frame defined within the duration of each visual element of the visual element sequence,
Utterance power calculation means for calculating utterance power of phonemes corresponding to visual elements of key frames in the key frame sequence from the utterance data;
For each key frame in the key frame sequence, a predetermined value is set such that the smaller the average utterance power calculated by the utterance power calculation means for the duration of the visual element including the key frame, the smaller the blend rate. Blend rate adjustment means by utterance power to adjust the blend rate by function,
A lip sync animation creating apparatus, comprising: blend processing means for creating an animation of a face image by blending between key frames based on a visual element sequence whose blend ratio has been adjusted by the blend ratio adjusting means. A key frame sequence whose blend rate is adjusted by the blend rate adjusting means based on the speech power is received, and a vertex constituting a face model corresponding to a visual element of each key frame included in the key frame sequence and an adjacent key frame A change speed calculating means for calculating a change speed between the vertices constituting the face model corresponding to the visual element;
Of the key frames included in the key frame sequence whose blend rate is adjusted by the blend rate adjusting unit based on the speech power, the change rate calculated by the change rate calculating unit is higher than a predetermined threshold value. A blend rate adjusting means by vertex speed for updating the blend rate using a predetermined function such that the blend rate becomes a smaller value for a large key frame,
The lip sync animation according to claim 7, wherein the blend processing unit creates an animation of a face image by blending between key frames based on a key frame sequence in which the blend rate is adjusted by the blend rate adjusting unit based on the vertex speed. Creation device. Motion vector calculating means for calculating a motion vector between feature points constituting the face model for all combinations of two face models selected from the plurality of face models;
The feature points of the two face models are clustered by a predetermined clustering method for the motion vector calculated by the motion vector calculating means, and a representative vector of each cluster is calculated to create a clustered face model Means for
Clustered face model storage means for storing the clustered face model;
The lip sync animation creating device further includes:
Clustering corresponding to a combination of a visual element of the key frame and a visual element of an adjacent key frame out of each key frame is received by the key frame sequence whose blend ratio is adjusted by the blend rate adjusting means by the speech power The calculation of the change speed for reading the combination of the face models from the clustered face model storage means and calculating the change speed between the key frames of the feature points belonging to each cluster using the representative vector of the cluster Means,
Among the key frames included in the key frame sequence, the blend rate of a key frame whose speed of change calculated by the speed of change calculating unit is larger than a predetermined threshold is set to a smaller value. A blend rate adjusting means based on the vertex speed for updating the blend rate using a predetermined function such that
The lip sync animation according to claim 7, wherein the blend processing unit creates an animation of a face image by blending between key frames based on a key frame sequence in which the blend rate is adjusted by the blend rate adjusting unit based on the vertex speed. Creation device. Lip sync animation from input speech data using statistical acoustic models prepared in advance, mapping definitions between phonemes and visual elements prepared in advance, and face models of multiple facial images prepared in advance A lip-sync animation creation device for creating a speech, the transcription for the utterance data is available,
Using the statistical acoustic model, the mapping definition, and the transcription, the phoneme sequence and the corresponding visual element included in the utterance data are obtained, and a visual element sequence with a duration is given a default blend rate. Including a visual element sequence creation means for creating
A key frame is defined during the duration of each visual element in the visual element sequence, and a key frame sequence is defined by these key frames,
Calculates the speed of change between the vertices constituting the face model corresponding to the visual element of each key frame included in the key frame sequence and the vertices constituting the face model corresponding to the visual element of the adjacent key frame. A speed of change calculation means for
Among the key frames included in the key frame sequence, the blend rate of a key frame whose speed of change calculated by the speed of change calculating unit is larger than a predetermined threshold is set to a smaller value. A blend rate adjusting means based on the vertex speed for updating the blend rate using a predetermined function as follows:
A lip sync animation creating apparatus, comprising: blend processing means for creating an animation of a face image by blending between key frames based on the key frame sequence whose blend rate is adjusted by the blend rate adjusting means based on the vertex speed. Lip sync animation from input speech data using statistical acoustic models prepared in advance, mapping definitions between phonemes and visual elements prepared in advance, and face models of multiple facial images prepared in advance A lip-sync animation creation device for creating a speech, the transcription for the utterance data is available,
Motion vector calculating means for calculating a motion vector between feature points constituting the face model for all combinations of two face models selected from the plurality of face models;
The feature points of the two face models are clustered by a predetermined clustering method for the motion vector calculated by the motion vector calculating means, and a representative vector of each cluster is calculated to create a clustered face model Means for
Clustered face model storage means for storing the clustered face model;
Using the statistical acoustic model, the mapping definition, and the transcription, the phoneme sequence and the corresponding visual element included in the utterance data are obtained, and a key frame sequence with a duration is given a default blend rate A key frame sequence creating means for creating
A key frame is defined during the duration of each visual element in the visual element sequence, and a key frame sequence is defined by these key frames,
The clustered face model storage means that receives the key frame sequence and stores a combination of clustered face models corresponding to a combination of a visual element of the key frame and a visual element of an adjacent key frame among the key frames. And a change speed calculation means for calculating the speed of change between key frames of feature points belonging to each cluster using a representative vector of the cluster, and
Among the key frames included in the key frame sequence, the blend rate of a key frame whose speed of change calculated by the speed of change calculating unit is larger than a predetermined threshold is set to a smaller value. A blend rate adjusting means based on the vertex speed for updating the blend rate using a predetermined function as follows:
A lip sync animation creating apparatus, comprising: blend processing means for creating an animation of a face image by blending between key frames based on the key frame sequence whose blend rate is adjusted by the blend rate adjusting means based on the vertex speed. Of the key frames included in the key frame sequence output by the visual element sequence creating means, the end position of the continuation length of the key frame immediately before the key frame to which the visual element corresponding to the blank phoneme is assigned is indicated in the key frame. Utterance end correction means for correcting the utterance end position by moving to the position after the maximum point of the utterance power sequence of the utterance data and within the duration of the key frame,
12. The lip sync animation creating apparatus according to claim 1, wherein the key frame deletion unit receives as input a key frame sequence whose utterance end is corrected by the utterance end correction unit. The utterance end correction means includes:
A first value that gives the maximum value of the utterance power of the key frame immediately before the key frame to which the visual element corresponding to the blank phoneme is assigned among the key frames included in the key frame sequence output by the visual element sequence creating means. Means for detecting the time;
Means for detecting a second time after the first time and before the end time of the key frame to be processed, a second time when the utterance power decreases from the maximum value of the utterance power by a predetermined rate;
13. The lip sync animation creating apparatus according to claim 12, further comprising means for correcting the key frame so as to move the end position of the key frame to be processed to the second time. The key frame creation means, when creating the key frame sequence, selects any one of the frames at the first frame rate as a key frame,
The lip sync animation creating device is further connected to receive an input designating a second frame rate smaller than the first frame rate and a key frame sequence output by the key frame deleting means, Frame rate conversion means for converting a key frame sequence output by the key frame deletion means into a key frame sequence of the second frame rate;
The frame rate converting means includes a key frame having a start edge within the continuation length of the key frame in the key frame sequence output from the key frame deleting means for each key frame of the key frame sequence at the second frame rate. Assign one of the visual elements assigned to,
The blend processing means includes means for creating an animation of a face image by blending between key frames based on the key frame sequence whose frame rate is converted by the frame rate conversion means. The lip sync animation creating apparatus according to any one of claims 13 to 13. The frame rate conversion means allocates a visual element so that a visual element allocated to each key frame of the key frame sequence of the second frame rate is different from a visual element allocated to the immediately preceding key frame. 14. The lip sync animation creating apparatus according to 14. The blend processing means has a function of creating an image for each frame at a third frame rate higher than the second key frame rate when creating an animation from the key frame sequence of the second frame rate. And having a function of generating an image of a frame between adjacent key frames by interpolation between adjacent key frames,
The lip sync animation creating apparatus further includes, for each key frame in the key frame sequence of the second frame rate output from the frame rate conversion means, a key frame and a key frame immediately after the key frame. 16. The lip sync animation creating apparatus according to claim 14, further comprising key frame copy means for copying the same key frame as the key frame at a frame position between them. The key frame copy means, for each key frame in the key frame sequence of the second frame rate output from the frame rate conversion means, at the frame position immediately before the key frame immediately after the key frame. 17. The lip sync animation creation device of claim 16, comprising means for copying the same key frame as the frame. 18. The lip sync animation creating apparatus according to claim 1, further comprising a face model storage unit for storing a face model of the plurality of face images. The pre-prepared phonemes include predetermined standard phonemes and general phonemes other than the standard phonemes,
The face models of the plurality of face images include a standard visual element model composed of a face model corresponding to the standard phoneme, and a general visual element model composed of a face model corresponding to the general phoneme,
The lip-sync animation creating apparatus further includes an actual measurement value of a three-dimensional position of a feature point of a face image of a speaker when speaking a corresponding phoneme, which is classified in advance corresponding to the phoneme prepared in advance. The lip sync animation creating apparatus according to claim 18, further comprising: general visual element generation means for generating the general visual element model using the captured data and the standard visual element model. The general visual element generation means is a linear sum of the capture data corresponding to the standard phonemes, and has a coefficient equal to the number of the standard phonemes for approximating the capture data corresponding to the general phonemes. Coefficient calculation means for calculating the approximation error to be minimum;
The general visual element model is calculated by a linear sum of the standard visual element models using the coefficients calculated by the coefficient calculating means for general phonemes corresponding to the general visual element model, and corresponds to the standard visual element model. The lip sync animation creating apparatus according to claim 19, further comprising: linear sum calculation means for storing the face model storage means in association with a general phoneme. A computer program that, when executed by a computer, causes the computer to function as the lip-sync animation creation device according to any one of claims 1 to 20.

Images