JP4543263B2 - Animation data creation device and animation data creation program - Google Patents

Description

  The present invention relates to an animation data creation technique, and more particularly to an animation data creation apparatus and program for creating an animation of a face image synchronized with voice from voice using a face model prepared in advance.

  Computer graphics (CG) are often used to create animation works, so that animations that require advanced skills of creators with conventional cell animations can be realized by simple tasks. became. Among the techniques using CG, for example, there is a technique for producing an animation using a three-dimensional model. In this technique, the shape, position, direction, etc. of an object are defined by polygons in a virtual space in each frame of the animation. Then, based on the definition, the images of the object are synthesized and an animation is constructed from the images. Once the shape of an object is defined, images from all viewpoints can be combined any number of times for that shape.

  By transforming an object into an image for each frame, it is possible to express changes in the character's facial expression. By preparing a separate voice as the character's voice and changing the mouth shape and facial expression of the character according to the voice, it is possible to produce an animation as if the character is speaking. In this specification, changing the shape and expression of a character's mouth in accordance with the voice is called “lip sync”. In this specification, an animation realized by lip sync is referred to as “lip sync animation”.

  In order to realize the lip sync, the voice of the character and the facial expression of the character expressed by the image of each frame must be synchronized. Conventionally used techniques for realizing lip sync are classified into the following two methods. One method is a method (after-recording: so-called “after-recording”) in which audio is recorded later in accordance with a previously produced video. Another method is a method in which audio is recorded first and a video is produced later according to the audio (pre-recording: this is hereinafter referred to as “pre-recording”). In post-recording, animation creators create images for each frame while predicting changes in the facial expression of the character being uttered. The speaker (or voice actor) who is in charge of the character's voice utters the speech by adjusting the timing while watching the expression of the character on the animation. On the other hand, in Pre-Reco, the speaker speaks freely. The producer creates an image of each frame while adjusting the facial expression according to the sound.

  As a technique for generating lip-sync animation using CG, Non-Patent Document 1 described later is recorded simultaneously with recorded audio data obtained by recording audio during utterance and when the recorded audio data is recorded. A statistical probability model creating apparatus for creating a statistical probability model for creating a lip sync animation from a data set composed of motion capture data relating to a plurality of feature points of a speaker's face is disclosed. This statistical probability model is a model that gives the probability of the position of each feature point with respect to an input phoneme label string or visual element string label.

  In the present specification, the “visual element” refers to the basic shape of the face (mainly the mouth), like the phoneme. There are a plurality of visual elements, and they are distinguished by names that identify visual elements. In this specification, the name of the visual element is referred to as a visual element label.

  By using this statistical probability model, it is possible to estimate a sequence of feature point position data having the highest likelihood for a phoneme label string or a visual element label string obtained from the input speech. A trajectory of the feature points of the face model synchronized with the input speech, that is, a wire frame model for each frame of the animation of the face image is obtained from the sequence of the estimated feature point position data. An animated image can be obtained by rendering to a wireframe model at each frame.

According to the disclosure of Non-Patent Document 1, not only the position data of facial feature points but also the speed and acceleration are added to the parameters for model learning when learning the statistical probability model, and only the position data is used. Compared to the case, it is possible to obtain a face animation that moves more naturally.
T. T. Yotsukura et al., "Lip-sync animation from HMM using dynamic features", ACM SIGGRAPH PH 2006 Proceedings CD, July 30, 2006 (T. Yotsukura et al., "Lip-sync Animation from HMM Using Dynamic Features", ACM SIGGRAPH 2006, 30 July 2006, Boston, Massachusetts)

  The method according to Non-Patent Document 1 described above is effective for creating a face animation having a smoother and more natural motion than when only static data such as position data is used. However, since dynamic data of feature points is used for model learning, the number of parameters during model learning is three times that in the case of using only static data. Therefore, there is a problem that model learning takes time. In particular, if the number of feature points is increased in order to create a more precise animation, the model learning time will increase accordingly.

  Further, since the position data of the facial feature points is estimated by statistical processing using an HMM (Hidden Markov Model), the estimated positional data is slightly shifted compared to the actual movement of the facial feature points. There is a problem that occurs.

  Therefore, there is a demand for a technology that can prepare for animation creation in a shorter time, and that can create a lip sync animation that naturally reflects the movement of the face and reflects the actual movement of the face.

  Therefore, an object of the present invention is to provide an animation data creation apparatus capable of creating a lip-sync animation capable of realizing a natural movement that well reflects an actual facial movement, and an animation data creation apparatus that can be prepared in a short time for animation creation. Is to provide a program.

  An animation data creation program according to a first embodiment of the present invention is a computer that includes a first storage unit that stores a visual element corpus, and that moves according to audio data based on input audio data. An animation data creation program for creating face animation data including The visual element corpus includes a plurality of visual element units created from an image of a face during speech with speech. Each visual element unit is a continuation of the phoneme corresponding to the visual element unit obtained from the visual element label, the motion data indicating the movement of the face corresponding to the visual element unit, and the sound corresponding to the visual element unit. Prosodic information including length. This program causes the computer to function as first conversion means for converting the voice data into a phoneme data string that identifies the phonemes represented by the voice data. The phoneme data string includes phoneme data including a phoneme label and prosodic information including the duration of the phoneme portion in the speech data. The program further converts each of the phoneme labels included in the phoneme data in the phoneme data sequence output by the first conversion means into a corresponding visual element label, thereby outputting a visual element data sequence. The computer functions as the second conversion means. The visual element data string output from the second conversion means includes a visual element label and a continuation length of at least the phoneme corresponding to the visual element data obtained from the portion corresponding to the visual element data in the audio data. Visual elementary data consisting of prosodic information is included. The program further includes, for each of the visual element data included in the visual element data string, a visual element label that is the same as the visual element label included in the visual element data in the visual element unit in the visual element corpus, and By using an evaluation function that evaluates the similarity of speech based on the prosodic information included in the visual elementary data and the prosodic information included in each visual elementary included in the visual elementary corpus, the speech most similar to the speech included in the visual elementary data is obtained. A first selection means for selecting a visual element unit evaluated to have from the visual element corpus, and motion data included in the visual element unit selected by the first selection means in accordance with the order of the visual element data string. By connecting on the time axis, the computer is used as a connection means for creating animation data of the mouth corresponding to the input audio data. It is allowed to function.

  The visual element corpus is stored in the first storage means in advance. The visual element corpus includes a plurality of visual element units created from an image of a face during speech with speech. The motion data contained in the visual element unit reflects the actual facial motion at the time of speech. The first conversion means converts the input voice data into a phoneme data string. The second conversion means converts the phoneme label included in the phoneme data string into a corresponding visual element label and outputs it as a visual element data string. The first selection means selects, from the visual element corpus, the visual element unit evaluated by the evaluation function as having the sound most similar to the sound included in the visual element data. The connecting means connects the motion data of the visual element units thus selected on the time axis to create animation data.

  The motion data used when creating the animation data is obtained from actual facial motion. Therefore, when they are connected, the motion of the face animation obtained at least in the portion corresponding to each visual element data becomes a natural one that well reflects the actual facial motion. The creation of the visual element corpus does not require learning processing using a large number of data as described in Non-Patent Document 1. Therefore, it is possible to provide an animation data creation program capable of creating a lip-sync animation that can be prepared in a short time and can realize a natural motion that well reflects the actual facial motion.

  Preferably, the prosodic information of the sound included in each of the visual element units included in the visual element corpus includes the average power of the sound during the duration in addition to the duration of the sound corresponding to the visual element unit, The conversion means 1 includes means for converting speech data into a phoneme data string, and the phoneme data string includes phoneme data including a phoneme label, a continuation length and an average power of the phoneme part in the speech data. The first selection means includes, for each of the visual element data included in the visual element data string, a visual element label that is the same as the visual element label included in the visual element data in the visual element unit in the visual element corpus. For each of the visual element units possessed, the audio similarity is determined by the duration and average power included in the visual elementary data and the duration and average power of the visual elementary unit. An evaluation means for evaluating the value of the evaluation function to be evaluated, and a visual element unit evaluated by the evaluation means to have the voice most similar to the voice included in the visual elementary data is selected from the visual elementary corpus Second selection means.

  The average power of the voice as well as the duration is used for the evaluation in the selection of the visual unit. The movement of each part of the face is affected by the loudness of the voice when speaking. Therefore, by evaluating the visual element unit to be selected using the average power of the sound in this way, it is possible to select a visual element unit that does not have a large discontinuity in the movement of each part of the face.

  As a result, it is possible to provide an animation data creation program capable of creating a lip-sync animation that can be prepared for animation creation in a short time and can realize a natural and smooth motion that well reflects the actual facial motion.

  More preferably, the computer further includes second storage means for storing a phoneme-visual element conversion table storing a correspondence relationship between phoneme labels and visual element labels. The second conversion means converts each of the phoneme labels included in the phoneme data of the phoneme data string output by the first conversion means into a corresponding visual element label by referring to the phoneme-visual element conversion table. Means for outputting a visual element data string.

  A phoneme data string is obtained using an established technique of converting voice data into a phoneme data string, and then a phoneme label is converted into a corresponding visual element label. Therefore, a system can be constructed efficiently using existing technology.

  More preferably, the number of visual element labels obtained by the conversion by the second conversion means is smaller than the number of phoneme labels output by the first conversion means.

  Compared with speech, the number of visual elements may be small. Thus, the processing can be stabilized by making the number of visual element labels smaller than the number of phoneme labels in this way.

  Each visual element unit of the visual element corpus includes a visual element label of a first number of visual element units preceding each visual element when a plurality of visual element units are created from a face image during speech with speech, And a visual element label of a second number of visual element units following each visual element. The visual element labels of the preceding first number of visual element units, the visual element labels of each visual element unit, and the visual element labels of the subsequent second number of visual element units constitute a set of visual element labels. To do. The second conversion means includes, for each phoneme data in the phoneme data string output by the first conversion means, the phoneme label included in the phoneme data and the first number of phoneme data before that. Means for converting each of the phoneme labels and the phoneme labels included in the subsequent second number of phoneme data into corresponding visual element labels and combining them in the order of the phoneme data to create a set of visual element labels And for each of the phoneme data in the phoneme data string output from the first converter, the phoneme label included in the phoneme data in the phoneme data string output from the first converter is set as a set of visual element labels. Means for creating and outputting visual label data by replacing with a set of visual labels obtained by the means for creating. The first selecting means selects, for each visual element data included in the visual element data string, a set of visual element labels that is the same as the set of visual element labels included in the visual element data to be processed in the visual element corpus. The visual element data is included in the visual element data by an evaluation function that evaluates the similarity of speech based on the prosodic information included in the visual element data to be processed and the prosodic information included in each visual element unit included in the visual element corpus. A second selection means for selecting from the visual element corpus the visual element unit that is evaluated to have the sound most similar to the sound to be recorded.

  By replacing the visual element label with the group of visual element labels in this way, the visual element unit can be selected in consideration of the shape of the face before and after the speech corresponding to the visual element unit. Therefore, it is possible to provide an animation data creation program capable of creating a lip sync animation that can realize a natural and smooth motion in which the actual facial motion is reflected in consideration of the shape of the previous and subsequent faces.

  The first number may be 1, and the second number may be 1.

  Preferably, the second selection means includes, for each of the visual element data included in the visual element data string, the same visual element label as the set of visual element labels included in the visual element data to be processed in the visual element corpus. And a visualizing unit for determining whether or not there is a visual element unit having the same set of visual element labels included in the visual element data to be processed in the visual element corpus. In response to determining that there is a visual elementary unit having a set of elementary labels, for each of the visual elementary units, the prosodic information included in the visual elementary data and each visual elementary unit included in the visual elementary corpus The first calculation means for calculating the value of the evaluation function for evaluating the similarity of speech based on the prosodic information possessed by and the determination means, the visual elementary data is converted into the visual elementary data by the determination means. In response to determining that there is no visual element unit having the same visual element label set as the set of visual element labels to be processed, out of the set of visual element labels of the visual element data to be processed, A visual element unit having a set of visual element labels whose positions and contents match the part of the visual element corpus, based on only a partial visual element label including a visual element label of visual element data. A second means for calculating a value of the evaluation function between the means for selecting and the prosody information included in the visual element data to be processed for each of the visual element units selected by the means for selecting Calculating means and means for selecting a visual element unit having the smallest value of the evaluation function calculated by the first calculating means or the second calculating means.

  When trying to select from the visual element corpus a visual element unit that has a visual element label that matches the set of visual element labels including the preceding and subsequent visual element labels, the variation of visual elements included in the visual element corpus is sufficient. When it is not large, there may be no visual element unit that satisfies the condition. Therefore, in such a case, an appropriate visual element unit can be surely selected by selecting a visual element unit from the visual element corpus that matches only the first half or only the second half. .

  More preferably, the linking means is the motion data of two preceding visual element units on the time axis among the motion data included in the visual element unit selected by the selecting means, and the motion data of the preceding visual element unit is selected. The motion data of the visual element unit is connected on the time axis by adding the motion data of the last part and the motion data of the first part of the subsequent visual element unit with weighting according to time. Including weighted addition means.

  Visual element units selected from the visual element corpus are not usually recorded consecutively. Therefore, some discontinuity may occur in the movement of the face. Thus, the motion data of two consecutive visual element units can be smoothly connected by weighting and adding at the boundary portion. As a result, it is possible to provide an animation data creation program capable of creating a lip-sync animation that can be prepared for animation creation in a short time and can realize a natural and smooth motion that well reflects the actual facial motion.

  A recording medium according to the second aspect of the present invention is a computer-readable recording medium on which any one of the animation data creation programs described above is recorded.

  An animation data creating apparatus according to a third aspect of the present invention uses an visual element corpus including a plurality of triple visual element units, and an animation for creating animation data indicating facial motion corresponding to input audio data A data creation device. Each triple visual element unit includes a triple visual element label, a visual element duration corresponding to the triple visual element unit, an average power of speech spoken when the visual element was recorded, and the visual elementary element. And motion data of feature points of the speaker's face. The animation data creation apparatus performs phoneme conversion to create a phoneme data string including a phoneme label, a phoneme duration, and an average power when the phoneme is uttered by performing speech analysis on the input speech data. Means, a means for storing a table indicating a correspondence relationship between the phoneme label and the visual element label, and converting the phoneme label included in the phoneme data string into a corresponding visual element label by referring to the table, A first conversion means for creating a visual elementary data string and a visual elementary label of the preceding and following visual elementary data for each of the visual elementary data in the visual elementary data string output by the first converting means. To a triple visual element label in combination with the second conversion means for outputting the triple visual element data string, and the triple visual element data output by the second conversion means For each of the triple visual element data included in the visual element corpus, a triple visual element unit having a triple visual element label that matches the triple visual element label of the triple visual element data from the visual elementary corpus, the triple visual element unit having The evaluation function that evaluates the similarity between the duration and power and the duration and average power of the triplet visual element data has a duration and average power similar to the duration and average power of the triplet visual element data. The selection means for selecting the triple visual element unit to be evaluated, and the face motion data included in the triple visual element unit selected by the triple visual element unit selection means, based on the time series of the triple visual element data, And connecting means for creating facial animation data by connecting the above.

  A visual element corpus is created in advance. The motion data included in the visual element unit of the visual element corpus reflects the actual facial movement. The first conversion means converts the input voice data into a phoneme data string. The second conversion means converts the phoneme label included in the phoneme data string into a corresponding visual element label and outputs it as a visual element data string. The selecting means selects, from the visual element corpus, the visual element unit evaluated by the evaluation function as having the sound most similar to the sound included in the visual element data. The connecting means connects the motion data of the visual element units thus selected on the time axis according to the time series of the visual element data, and creates animation data.

  The motion data used when creating the animation data is obtained from actual facial motion. Therefore, when they are connected, the movement of the face animation obtained at the portion corresponding to each visual element data becomes a natural one that well reflects the actual movement of the face. The creation of the visual element corpus does not require learning processing using a large number of data as described in Non-Patent Document 1. Therefore, it is possible to provide an animation data creation program capable of creating a lip-sync animation that can be prepared in a short time and can realize a natural motion that well reflects the actual facial motion.

  Hereinafter, a lip sync animation creating apparatus according to an embodiment of the present invention will be described. As will be described later, this lip sync animation creating apparatus is realized by computer hardware, a program executed by the computer hardware, and data such as an acoustic model stored in a storage device of the computer.

  First, terms used in the following description will be described.

  “Visual element” is a translation of “viseme” in English. Also called “viseme”. A visual element is a set of information representing the shape of a basic face (especially the mouth) existing in the movement of a face, like a phoneme in speech.

  The “visual element label” refers to a name given to each visual element in order to identify the visual element. Used in the same way as “phoneme labels” in phonemes.

  The “visual element corpus” refers to a database in which the movement of the speaker's face when speaking is recorded by a motion capture device and divided and stored for each visual element. In the present embodiment, the visual element corpus includes a plurality of visual element units. Each visual element unit corresponds to the time series data of the position vector of the face feature point, the name of the visual element, the time information of the portion corresponding to each visual element in the time vector data of the position vector, and each visual element. And the power of the voice of the part to be In the present embodiment, the visual data recorded first is also attached to the visual element corpus. This is called a “speech-visual elementary corpus”.

  “Visual element data” refers to data that is obtained from input speech and serves as a reference for selecting a visual element from a visual element corpus. In this embodiment, the visual element data includes a visual element label of a visual element to be selected, its duration, and the average power of input speech corresponding to the visual element. The duration of the visual element is also obtained from the duration of the phoneme of the input speech corresponding to the visual element.

  “Triple visual element label” means a visual element label of a certain visual element, a visual element label of the visual element immediately before that visual element, and a visual element label of the visual element immediately after that visual element on the time axis. It means a combination according to the order. In the present embodiment, each visual element unit in the visual element corpus is labeled with this triple visual element. These are referred to herein as triple visual element units.

[Constitution]
Hereinafter, a functional configuration of a lip sync animation creating apparatus realized by a program according to an embodiment of the present invention will be described. FIG. 1 shows a block diagram of the lip sync animation creating apparatus 40. Referring to FIG. 1, the lip-sync animation creation device 40 records the movement of the facial feature point of the speaker 50 when speaking a predetermined text together with the voice, and creates a voice-visual elementary corpus. The recording system 60 for recording, the speech-visual element corpus storage unit 62 for storing the speech-visual element corpus created by the recording system 60, and the input speech data 42 are operated in synchronization with the speech data 42. , An animation data synthesis device 44 for synthesizing a motion vector sequence of facial feature points as animation data, and an animation data storage unit 46 for storing animation data synthesized by the animation data synthesis device 44.

  The audio-visual element corpus stored in the audio-visual element corpus storage unit 62 includes a triple visual element unit sequence obtained from the video of the speaker 50 at the time of speaking.

  The lip-sync animation creation device 40 further reads out the animation data stored in the animation data storage unit 46 at the time of creating the actual animation, and applies this animation to the face model of the animation character consisting of a wire frame prepared in advance. By applying the data, time series data of the face model that moves in synchronization with the input audio data 42 is generated, and further, the face texture is applied to the face model for rendering, thereby rendering a predetermined frame / second. And an animation storage unit 98 for storing the animation created by the animation creating device 48 together with the audio data 42.

  The lip sync animation creating apparatus 40 further includes an animation reading unit 100 for reading an animation stored in the animation storage unit 98 and writing it in a frame memory (not shown) at a predetermined frame rate when displaying the animation, and the animation reading unit 100. And a display unit 52 for reproducing and displaying the animation written in the frame memory together with the sound.

  FIG. 2 shows the configuration of the recording system 60. Referring to FIG. 2, the recording system 60 includes a recording / recording system 112 for recording the voice of the speaker 50 and a moving image of the speaker 50 at the time of speaking, and the face of the speaker 50 at the time of speaking. Measured by a motion capture (Motion Capture; hereinafter referred to as “MoCap”) system 114 for measuring the position of each part and its trajectory, audio / video data 116 recorded by the recording / recording system 112, and the MoCap system 114. From the generated data (hereinafter, this data is referred to as “MoCap data”) 118, voice data, three-dimensional motion vector of each part of the speaker's face at the time of utterance, visual element label, visual element continuation length , And a data set 120 consisting of a series of average power of the voice when the visual element is uttered, Audio Satoshimoto corpus storage unit 62 - and a data set creation unit 122 for storing a viseme corpus. Note that the three-dimensional data of the feature points of the speaker's face is processed so as to be a motion vector from which the head motion is removed, as will be described later. In this specification, this processing is called normalization processing, and the three-dimensional motion vector series of the facial feature points after normalization is called a face parameter.

  The recording / recording system 112 receives microphones 130A and 130B for receiving voices uttered by the speaker 50 and converting them into voice signals, and captures a moving image of the speaker 50, and the video signals and the microphones 130A and 130B. And a camcorder 132 for simultaneously recording audio signals and generating audio / moving image data 116.

  The camcorder 132 has a function of supplying the time code 134 to the MoCap system 114. The camcorder 132 has a function of converting an audio signal and a video signal into data in a predetermined format, and adding the same time code as the time code 134 to record it on a recording medium (not shown).

  The MoCap system 114 according to the present embodiment includes an optical system that measures the position of a measurement target using the reflected light of a highly recursive optical reflection marker (hereinafter simply referred to as “marker”). The MoCap system 114 includes a plurality of infrared cameras 136 </ b> A for capturing images of infrared reflected light from markers attached to a plurality of predetermined parts of the head of the speaker 50 for each frame at predetermined time intervals. .., 136F and a data processing device 138 for measuring the position of each marker for each frame based on the video signals from the infrared cameras 136A,..., 136F, adding the time code 134 from the camcorder 132, and outputting it. Including.

  In FIG. 3, the example of the mounting position of the marker with which the speaker's 50 head 110 is mounted | worn is shown typically. Referring to FIG. 3, markers are attached to a large number of locations 160 on the face, neck, and ears close to the head 110 of the speaker 50. The marker has a hemispherical shape or a spherical shape, and its surface is processed to retroreflect light. The size of the marker is about several millimeters in diameter. In order to enrich the speech-visual corpus 62, it is necessary to perform measurement over a plurality of days or a plurality of speakers 50. For this reason, the mounting order of the markers is determined in advance, and the position determined by the relative position with the characteristic position of the facial organ or the mounted marker is determined in advance as the mounting position. The mounting position thus determined is referred to as a “feature point” in this specification.

  Due to the physical structure of the face, there are places on the face of the speaker 50 that follow the movement of the head itself, but are hardly affected by changes in the expression of the speaker 50. For example, the temples 160A and 160B and the tip 160C of the nose have such characteristics. In this embodiment, such a location is determined in advance as a feature point. Hereinafter, such a feature point is referred to as a fixed point. In the motion capture, the three-dimensional position of the facial feature point is measured, and the change in the position includes a change due to the movement of the head 110 of the speaker 50 itself. In order to obtain the movement of the face, it is necessary to subtract the movement of the head from the position data of each feature point. This process is called normalization. Details thereof will be described later. The fixed point is used in the normalization process. For the normalization process, it is desirable to determine four or more fixed points.

  Referring again to FIG. 2, the data processing device 138 generates MoCap data 118 by collecting the measurement data (hereinafter referred to as “marker data”) at the positions of the markers for each frame, and the data set creation device 122. Output to. A commercially available optical MoCap system can be used for the MoCap system 114. Since the functions and operations of the infrared camera and the data processing device in the commercially available optical MoCap system are well known, detailed description thereof will not be repeated here.

  The data set creation device 122 reads out the audio / moving image storage unit 140 for capturing and storing the audio / moving image data 116, and the audio / moving image data 116 stored in the audio / moving image storage unit 140, and performs triple visual element data. A triple visual element data sequence creation unit 144 for creating and outputting the sequence 124, a MoCap data storage unit 142 for capturing and storing the MoCap data 118, and the MoCap data stored in the MoCap data storage unit 142. The normalization processing unit 146 for normalizing and converting the MoCap data 152 into the face parameter series 126 of each feature point of the face, the triple visual element data string 124 from the triple visual element data string creating unit 144, and The face parameter series 126 from the normalization processing unit 146 is synchronized using their time stamps. Thereby generating a data set 120 by binding to, voice - and a coupling portion 148 for causing stored as viseme corpus - voice viseme corpus storage unit 62.

  The normalization processing unit 146 has a function of generating a face parameter of the frame by converting each marker data of the frame so that the position change of the fixed point becomes 0 in each frame of the MoCap data 152. Have. In this embodiment, affine transformation is used for this transformation.

  The marker data in the MoCap data 152 of the frame at time t = 0 is P = <Px, Py, Pz, 1> in the homogeneous coordinate system, and the marker data at time t ≠ 0 is P ′ = <P′x, P′y. , P′z, 1>, the relationship between the marker data P and the marker data P ′ is expressed by the following equation (1) using the affine matrix M.

顔パラメータの系列１２６の各フレームにおいて不動点の位置データがすべて同じ値となれば、不動点の位置変化が０になり、それ以外の特徴点の位置を不動点の位置を基準として正規化できる。そこで、本実施の形態では、フレームごとに、ｔ＝０のフレームにおける各不動点のマーカデータと、処理対象のフレームにおける当該不動点のマーカデータとから、当該フレームにおけるアフィン行列Ｍを算出する。このアフィン行列Ｍを用いて、各マーカデータをアフィン変換する。変換後のマーカデータはそれぞれ、ｔ＝０での頭の位置のまま発話を行なった状態での顔の特徴量の位置を表すものとなる。

If the fixed point position data in all frames of the face parameter series 126 have the same value, the fixed point position change becomes zero, and the positions of other feature points can be normalized with reference to the fixed point position. . Therefore, in the present embodiment, for each frame, the affine matrix M in the corresponding frame is calculated from the marker data of each fixed point in the frame at t = 0 and the marker data of the fixed point in the processing target frame. Using this affine matrix M, each marker data is affine transformed. Each of the converted marker data represents the position of the facial feature amount in a state where the speech is performed with the head position at t = 0.

  Further, in the present embodiment, by subtracting the marker data of each feature point obtained from the face image of the expressionless speaker from the marker data of each feature point obtained by the above normalization, The position of the feature point is represented by a motion vector. By doing so, the following processing can be performed when creating an animation of a face model.

Referring to FIG. 4, it is assumed that a face model 170 of an animation character is prepared in advance. When a face image animation 172 composed of three consecutive frames 180, 182, and 184 is created for this face model 170, the motion vector obtained by the above-described processing is added to the marker data of each feature point of the face model 170. V ₁₈₀ , V ₁₈₂ and V ₁₈₄ are added. By this processing, the position of each feature point of the face model in each of the three frames 180, 182 and 184 is obtained. Actually, the face model is given by a wire frame, and the position of the feature point does not necessarily match the position of the node of the wire frame. Therefore, it is necessary to map the feature point to the node of the face model 170. Details of the deformation of the face model will be described later.

  Details of the triple visual element data string creation unit 144 shown in FIG. 2 will be described with reference to FIG. Referring to FIG. 5, triple visual element data string creation unit 144 reads audio / video data 116 from audio / video storage unit 140 and frames the audio with a predetermined frame length and frame interval for acoustic processing. A framing processor 200, a feature extracting unit 201 for extracting a feature amount 230 used in Viterbi alignment (to be described later) from audio data of each frame output by the framing processor 200, and a speaker 50 (FIG. 1), an acoustic model storage unit 202 for storing a statistical acoustic model obtained by speech learning, and an utterance text storage unit 204 for storing utterance texts when recording the utterance data by the recording system 60; The acoustic model stored in the acoustic model storage unit 202 is extracted from the sequence of feature amounts output by the feature extraction unit 201. A sequence of phoneme data consisting of a label of each phoneme corresponding to the utterance text and its duration by viterbi alignment using the utterance text stored in the utterance text storage unit 204 and having the maximum likelihood (phoneme data A Viterbi alignment unit 206 for outputting the column 232), and a phoneme data sequence storage unit 208 for storing the phoneme data sequence 232 output by the Viterbi alignment unit 206. In the present embodiment, an acoustic model composed of an acoustic HMM is used.

  The triple visual element data sequence creation unit 144 further includes a phoneme-visual element conversion table storage unit 210 for storing a phoneme-visual element conversion table indicating the correspondence between phoneme labels and visual element labels, and a phoneme data sequence. The phoneme data string stored in the storage unit 208 is read out, and the phoneme label included in each phoneme data is referred to the phoneme-visual element conversion table stored in the phoneme-visual element conversion table storage unit 210 to correspond to the visual The phoneme-visual element conversion unit 212 for outputting the visual element data string 234 and the phoneme-visual element conversion unit 212 to output visual element data including the visual element label and its continuation length. Visual element data string storage unit 214 for storing the visual element data string 234 to be read, and the visual element data string stored in the visual element data string storage unit 214 are read out. The visual element label included in the data is a triple visual combination that combines the visual element label of the preceding visual element data, the visual element label of the visual element data to be processed, and the visual element label of the immediately subsequent visual element data in this order. A visual element-triple visual element conversion unit 216 for converting into a prime label and outputting as a triple visual element data string 236 and a triple visual element data string 236 output by the visual element-triple visual element conversion unit 216 are stored. And a triple visual element data string storage unit 218. When the phoneme-visual element conversion unit 212 converts the phoneme label into the visual element label, and the same visual element label continues, the phoneme-visual element conversion unit 212 combines them into one visual element data, and totals the continuation length. The phoneme-visual element conversion unit 212 further calculates the average power of speech corresponding to each visual element data, and assigns it to the visual element data as prosodic information.

  FIG. 6 shows an outline of processing performed by the Viterbi alignment unit 206. In the present embodiment, the feature extraction unit 201 calculates an MFCC (Mel Frequency Cepstrum Coefficient) as the feature amount 230 from each frame of speech, and gives it to the Viterbi alignment unit 206. The Viterbi alignment unit 206 uses a large number of phoneme HMMs stored in the acoustic model storage unit 202 and the utterance text stored in the utterance text storage unit 204, and has the highest likelihood as the division of the phoneme string corresponding to the utterance text. The speech is divided into phoneme strings according to a division method that increases, and a phoneme data string 232 composed of the label of each phoneme and its duration is output.

  FIG. 7 shows an example of the triple visual element data string 236 output from the visual element-triple visual element conversion unit 216 and stored in the triple visual element data string storage unit 218. As shown in FIG. 7, each of the triple visual element data includes a triple visual element label, a duration in milliseconds, and an average power of the voice over the entire duration. In FIG. 7, the symbol in the center of the triple visual element is the original visual element label of the visual element data. The visual element label of the immediately preceding visual element data is attached to the left side with the symbol “-”, and the visual immediately after that is attached to the right side with the symbol “+”. It is a visual elementary label of raw data. In the figure, “sil” represents a visual element label corresponding to a silent state, “sp” represents a visual element label corresponding to a short pose, and A, R, Y, etc. represent visual elements corresponding to predetermined phonemes, respectively. Indicates an elementary label.

  FIG. 8 shows an example of the phoneme-visual element conversion table stored in the phoneme-visual element conversion table storage unit 210 shown in FIG. Various configurations of the phoneme-visual element conversion table are conceivable. Basically, a phoneme-visual element conversion table associates a phoneme label with a visual element label indicating a mouth shape when the phoneme is pronounced. As shown in FIG. 8, in this embodiment, one visual element label is associated with one or more phoneme labels. This is because even if the sound being pronounced is different, the mouth shape may be very similar, and in such a case, it would be strange to create animations with the same mouth shape for different sounds. Not based on.

  In FIG. 8, “silB” represents the silent state immediately before the utterance, and “silE” represents the silent state immediately after the utterance.

  From the above, the configuration of the speech-visual element corpus stored in the speech-visual element corpus storage unit 62 is as shown in FIG. Referring to FIG. 9, the audio-visual element corpus includes audio waveform data 240, a motion vector sequence 242, and a triple visual element unit sequence 244. Both the audio waveform data 240 and the motion vector sequence 242 are given time codes. In this embodiment, the voice waveform data 240 is not used.

  Each unit in the triple visual element unit column 244 corresponds to the unit ID (identification number) for identifying the unit, the triple visual element label of the unit, the duration of the visual element of the unit, and the visual element. And the time indicating the start position in the motion vector sequence 242 of the motion vector corresponding to the visual element. In the present embodiment, the motion vector sequence is not included in the triple visual element unit, but the motion vector sequence belonging to the visual element unit is obtained by referring to the motion vector sequence 242 with the start position and the duration. It can be known where in the motion vector sequence 242.

  Referring to FIG. 1 again, the animation data synthesizing device 44 has the same function as the triple visual element data string creation unit 144 shown in FIG. 2 for creating a triple visual element data string from the input audio data 42. For each of the visual element data included in the triple visual element data string generation unit 80 and the triple visual element data string generated by the triple visual element data string generation unit 80, an animation synchronized with the input audio data 42 is generated. A triple visual element unit selection unit 82 for selecting the triple visual element unit evaluated to be optimal for creation from the speech-visual element corpus storage unit 62, and the triple selected by the triple visual element unit selection unit 82 By connecting the three-dimensional motion vectors of facial feature points included in the visual unit to each other along the time axis, And a triad viseme unit connecting portion 84 for creating Deployment data.

  FIG. 10 shows details of the configuration of the triple visual element data string creation unit 80. The triple visual element data string creation unit 80 has basically the same configuration as the triple visual element data string creation unit 144 shown in FIG.

  Referring to FIG. 10, triple visual element data sequence creation unit 80 includes a framing processing unit 280 for framing input audio data 42 with frames having a predetermined frame length and a predetermined frame interval, and a framing processing unit MFCC is extracted as a feature amount from each frame of speech data output by 280, and learning is performed using a feature amount extraction unit 282 for outputting a sequence of feature amounts and the voice of the speaker of the speech data 42. An acoustic model storage unit 284 for storing the acoustic model, an utterance text storage unit 286 for storing the utterance text of the input voice data 42, and a feature amount sequence extracted by the feature amount extraction unit 282 The acoustic model stored in the acoustic model storage unit 284 and the utterance text stored in the utterance text storage unit 286 are used. It was subjected to Viterbi alignment, and a Viterbi alignment unit 288 for outputting a sequence of phonemic data including phoneme label and duration thereof phoneme in accordance with spoken text (phoneme data strings).

  The triple visual element data string creation unit 80 further stores a phoneme-visual element conversion table storage for storing the same phoneme-visual element conversion table stored in the phoneme-visual element conversion table storage unit 210 shown in FIG. The phoneme labels of the phoneme data included in the phoneme data string output by the unit 290 and the Viterbi alignment unit 288 are referred to the phoneme-visual element conversion table stored in the phoneme-visual element conversion table storage unit 290. A phoneme-visual element conversion unit 292 for converting to a corresponding visual element label and outputting a visual element data sequence, and a visual element data for storing a visual element data sequence output by the phoneme-visual element conversion unit 292 The visual element data string stored in the column storage unit 293 and the visual element data string storage unit 293 is read out, and the visual element label is assigned to each of the visual element data. A visual element-triple visual element conversion unit 294 for outputting a triple visual element data string by replacing with a visual element label obtained by sequentially combining the visual element labels of the subsequent visual element data, and a visual element A triple visual element data string storage unit 295 for storing the triple visual element data string output by the triple visual element conversion unit 294; The triple visual element unit selection unit 82 shown in FIG. 1 reads the triple visual element data string 296 from the triple visual element data string storage unit 295.

  FIG. 11 shows the configuration of the triple visual element data string 296 passed from the triple visual element data string creation unit 80 shown in FIG. 1 to the triple visual element unit selection unit 82. Referring to FIG. 11, each of the triple visual element data included in this triple visual element data sequence 296 includes a sequence number indicating the order of visual elementary data in the input audio data 42, a triple visual element label, It includes the duration of the visual element and the average power of the audio corresponding to this visual element data.

  FIG. 12 shows a configuration of a triple visual element data string 300 output from the triple visual element unit selection unit 82. Referring to FIG. 12, the triple visual element data included in the triple visual element data string 300 has the same configuration as the triple visual element data string shown in FIG. 11, but is optimal for generating animation data. And a selection unit ID for identifying the visual element unit selected from the speech-visual element corpus storage unit 62. This selection unit ID corresponds to the “unit ID” at the left end of the triple visual element unit row 244 shown in FIG. If this unit ID exists, the audio-visual element corpus storage unit 62 is referred to, and data of “start time” and “continuation length” of the corresponding triplet visual element unit in the triplet visual element unit column 244 is used. A motion vector sequence belonging to this unit can be extracted from the motion vector sequence 242.

  FIG. 13 is a flowchart showing the control structure of a computer program for causing a computer to function as the triple visual element unit selection unit 82. Referring to FIG. 13, in this functional block, in step 310, the triple visual element data at the read pointer position is read from the triple visual element data string output by the triple visual element data string creation unit 80. In step 312, it is determined whether or not the read pointer position has reached the end position of the triple visual element data string to be read. If it has reached, the process is terminated. If the end position has not been reached, the process proceeds to step 314.

  In step 314, the speech-visual element corpus in which the visual element unit having the triple visual element label that matches the triple visual element label included in the triple visual element data read in step 310 is stored in the audio-visual element corpus storage unit 62. It is determined whether it exists in the inside. If such a visual element unit is present in the audio-visual corpus, the process proceeds to step 318, and if not, the process proceeds to step 316.

  In step 318, all triple visual element units found in step 314 are read from the speech-visual element corpus. Thereafter, the process proceeds to step 320.

  On the other hand, in step 316, among the triplet visual element labels included in the triplet visual element data read in step 310, two that match the first half pair visual element label or the second half pair visual element label. All triplet visual element units having the first half or the second half of the triplet visual element label are read from the speech-visual element corpus. Thereafter, the process proceeds to step 320.

  In step 320, the cost C is calculated for all of the triple visual element units read in step 316 or 318 by the following formula.

ただし、ＴＤ及びＴＰは、ステップ３１０で読んだ三つ組視覚素データに含まれる視覚素継続時間及び平均パワーであり、ＵＤ及びＵＰは、コスト計算の対象となっている三つ組視覚素ユニットに含まれる視覚素の継続時間及び平均パワーであり、ｗ_ＴＤ及びｗ_ＴＰはそれぞれ継続時間の差及び平均パワーの差に対して割当てられる重みである。重みｗ_ＴＤ及びｗ_ＴＰは、話者の相違などの条件によって異なるため、主観的テストによって決定する必要があるが、例えばｗ_ＴＤ＝ｗ_ＴＰとしてもよい。また、この状態からｗ_ＴＤ及びｗ_ＴＰの値を少しずつ変えることにより、これらの値の好ましい組合せを徐々に求めるようにしてもよい。

However, TD and TP are visual element duration and average power included in the triple visual element data read in step 310, and UD and UP are visual elements included in the triple visual element unit that is the object of cost calculation. The raw duration and average power, w _TD and w _TP are weights assigned to the duration difference and the average power difference, respectively. Since the weights w _TD and w _TP differ depending on conditions such as speaker differences, it is necessary to determine the weight w _TD and w _TP by a subjective test. For example, w _TD = w _TP may be used. Moreover, you may make it obtain | require gradually the preferable combination of these values by changing the value of _wTD and _wTP little by little from this state.

  The cost C calculated here is evaluated using the prosodic feature of the speech corresponding to the visual element, which is optimal as the triple visual element unit that gives the motion vector of the facial feature point with respect to the triple visual element data being processed. Is an evaluation function. As can be seen from the above formula, in the present embodiment, as a cost function, the absolute value of the difference between the durations of the visual elements (which is equal to the duration of the speech corresponding to the visual elements) and the visual elements are supported. The linear sum of the absolute values of the difference in the average power of the sound to be used is used. Various other cost functions are conceivable. When determining the structure of triple visual element data and triple visual element units, it is necessary to consider what information is used as a cost function and to store necessary data.

  In step 322, the minimum value of the cost calculated in step 320 is obtained, and the triple visual element unit giving the minimum value is selected. This triplet visual element unit is selected as giving the optimal motion vector sequence for the triplet visual element data being processed. Thereafter, the control returns to step 310 to execute processing for the next triplet visual element data.

  The triple visual element unit selection unit 82 shown in FIG. 1 can be realized by a computer program having a control structure corresponding to the flowchart shown in FIG.

  Next, the function of the triple visual element unit connecting portion 84 shown in FIG. 1 will be described. As described above, the triple visual element unit selection unit 82 shown in FIG. 1 selects the triple visual element unit sequence corresponding to the input audio data 42. If these triplet visual element unit sequences are connected as they are on the time axis, the position of each feature point may shift between units, or a gap or overlap between units may occur on the time axis. The image becomes unnatural. The triple visual element unit connecting unit 84 has a function of connecting the triple visual element units on the time axis while eliminating such positional shift of the feature points.

  FIG. 17 shows a motion vector connection method by the triple visual element unit connection unit 84. Referring to FIG. 17 (A), a trajectory 430 of a certain feature point M1 in a certain triplet visual element unit and a trajectory 432 of a corresponding feature point M1 ′ in a subsequent triplet visual element unit are obtained only at time T at both ends thereof. Duplicate. Then, the trajectory of the feature point at this time is calculated by smoothing according to the following formula, and a smooth trajectory 440 is generated.

ただし、Ｍは平滑化後の特徴点の動きベクトル、Ｍ_１及びＭ_１’はそれぞれ平滑化前の、先行及び後続する三つ組視覚素ユニットの特徴点の動きベクトル、ｔは重複区間Ｔの先頭からの経過時間を示す。三つ組視覚素ユニット連結部８４は、これ以外の区間では、その三つ組視覚素ユニットの動きベクトル列をそのまま出力する。

Where M is the motion vector of the feature point after smoothing, M ₁ and M ₁ ′ are the motion vectors of the feature points of the preceding and succeeding triple visual element units before smoothing, and t is the head of the overlap section T. Shows the elapsed time. The triple visual element unit connecting unit 84 outputs the motion vector sequence of the triple visual element unit as it is in other sections.

  If such a connection is made, the duration of each triplet visual element unit is substantially shortened by T. To prevent this, one end (for example, the rear end) of each triplet visual element unit is connected. Extend by T. By adopting the structure of the audio-visual element corpus storage unit 62 as shown in FIG. 9, such a triple visual element unit can be easily extended.

  The triple visual element unit connecting unit 84 performs such connection with respect to all feature points for all of the connecting parts in the triple visual element unit row output from the triple visual element unit selecting unit 82. As a result, a data series having motion vectors for all the facial feature points is obtained for each predetermined period. This data is called animation data in this specification. The animation data is stored in the animation data storage unit 46.

  The animation creation device 48 has a function of creating an animation from animation data stored in the animation data storage unit 46 and a face model of the animation character prepared in advance.

  Referring to FIG. 1, the animation creation device 48 includes a face model storage unit 90 for storing a face model of an animation character. In the present embodiment, the face model stored in the face model storage unit 90 uses a shape model that expresses a predetermined face shape in a stationary state by a large number of polygons (polygons). In order to create an animation using the animation data stored in the animation data storage unit 46 based on the face model, the face model is associated (mapped) with the position of the feature point corresponding to the animation data. Must be performed in advance. In the present embodiment, the position of the feature point is manually mapped to the face model, and the position of the feature point on the face model is indicated by a marker called “virtual marker”.

  FIG. 14 shows an example of the face model 330 and the virtual marker. With reference to FIG. 14, the sides of the polygons constituting the face model 330 (black lines constituting the sides of the triangle in FIG. 14) are called edges, and the intersections of the edges are called nodes in the face model 330. In FIG. 14, a virtual node mapping example is shown as a symbol 332 that combines the symbol O and the + mark.

  The face has cuts that do not constitute the face, such as the eyes, mouth, and nostrils. In general, these breaks are not modeled as part of the face model 330. That is, polygons are not defined for the cuts, or the cuts are defined as objects different from the face model 330. Therefore, the cut and the face are partitioned by the boundary edge. A boundary edge is an edge that is not shared by two polygons.

  Referring again to FIG. 1, the animation creation device 48 further reads out the data at the time corresponding to each frame of the animation from the animation data stored in the animation data storage unit 46 and stores it in the face model storage unit 90. A face model deforming unit 92 for deforming and outputting the face model according to the motion vector in the read animation data, and face texture data and illumination for creating character animation by rendering the face model Rendering data storage unit 94 for storing settings such as position and camera position, and rendering data stored in the rendering data storage unit 94 for the face model of each frame output by the face model deformation unit 92 Render using Output for each frame of the animation and a rendering unit 96 for storing the animation storage unit 98.

  The shape of the face represented by the face model 330 indicates the basic shape of the face of the animation character, and may be any created by the user. However, as described above, in order to give a facial expression to the face model 330 using the motion vector, it is necessary to define which part of the shape expressed by the face model 330 corresponds to the feature point. A virtual marker 332 indicates such a correspondence. The face model storage unit 90 shown in FIG. 1 stores not only the three-dimensional position data of each node of the face model, but also the positions of the virtual markers and the correspondence between the virtual markers and feature points.

  The face model deforming unit 92 deforms the face model stored in the face model storage unit 90 as follows. Basically, the face model deforming unit 92 performs the following process (referred to as marker labeling process) on all the nodes constituting the face model for each read animation data. That is, the face model deforming unit 92 selects, for each face model node, a virtual marker having the closest distance from the node based on the coordinates of the virtual marker. The face model deformation unit 92 determines whether or not the selected virtual marker is appropriate as a virtual marker associated with the node being processed. If it is appropriate, the selected marker is adopted as a marker corresponding to this node, and if it is inappropriate, it is rejected. Such processing is repeated, and a predetermined number n (for example, n = 3) of virtual markers is adopted for one node of the face model. In this specification, a virtual marker adopted for a certain node is called a “corresponding marker” of the node.

  In the present embodiment, the boundary edge of the face model is used as a reference when determining whether the selected marker is appropriate as a corresponding marker.

  When a corresponding marker is related to each node of the face model by this marker labeling process, the three-dimensional position coordinates of that node are obtained by interpolation of the motion vector value of the feature point corresponding to the corresponding marker obtained from the animation data. Is calculated. This calculation method will be described later.

  FIG. 15 is a flowchart showing a control structure of a marker labeling process program executed by the face model deforming unit 92. Referring to FIG. 15, in the marker labeling process, the process from step 342 to step 354 surrounded by step 340 </ b> A and step 340 </ b> B is executed for each node of face model 330.

  In step 342, the distance from the processing target node to the virtual marker is calculated. Furthermore, a list of virtual markers sorted in ascending order of the distance is displayed.

  In step 344, 0 is substituted into a variable i for controlling the following repetition and a variable j representing the number of markers employed as corresponding markers. In step 346, 1 is added to the variable i.

  In step 347, it is determined whether or not the value of the variable i exceeds the number Mmax of virtual markers. If the value of the variable i exceeds the number Mmax of virtual markers, an error occurs and the process is terminated. This occurs when all the virtual markers are examined, but less than three are used as corresponding markers. Normally, this is not the case, but such error handling is provided just in case. If the value of variable i is less than or equal to the number of virtual markers Mmax, control proceeds to step 348.

  In step 348, the line segment connecting the virtual marker (hereinafter referred to as “marker (i)”) existing at the position indicated by the variable i from the head of the list and the node to be processed is It is determined whether or not the boundary condition that the boundary edge is not crossed is satisfied. If the line segment crosses one of the boundary edges, the process returns to step 344. Otherwise, go to step 350.

  In step 350, the marker (i) at this point is designated as one of the corresponding markers of the processing target node. Information indicating the marker (i) is stored as marker / node correspondence information of the processing target node. Thereafter, the control proceeds to step 352. In step 352, 1 is added to the variable j. In step 354, it is determined whether or not the value of the variable j is 3. If the value of the variable j is 3, the process proceeds to step 340B. Otherwise, go to step 344.

  As described above, the line segment that connects the node to be processed and the virtual marker crosses the boundary edge of the face model is excluded from the virtual marker corresponding to the node to be processed. This is due to the following reason. For example, consider a case where a boundary edge exists between the upper lip and the lower lip. In this case, in the actual face, the positions corresponding to the node located on the upper lip and the node located on the lower lip move differently. Therefore, for example, when calculating the movement amount of the upper lip node, it is not appropriate to use the movement amount of the marker existing on the lower lip. Whether or not the line segment crosses a certain boundary edge is determined by, for example, whether the boundary edge is shared by two polygons constituting the face model or belongs to only one.

  FIG. 16 shows polygons around the lips and virtual markers in the face model 330. Hereinafter, a method for specifying the corresponding marker of a certain node will be described with reference to FIG. 16 using a specific example. Referring to FIG. 16, there are a large number of triangular polygons around the lips of face model 330 (see FIG. 14). Each polygon is surrounded by three edges. A boundary edge 400 exists between the upper lip and the lower lip. The boundary edge 400 corresponds to the boundary between the face model 330 and the cut or the outer edge of the face model 330. Therefore, edges other than the boundary edge are shared by the two polygons, but the edge corresponding to the boundary edge 400 is not shared.

  As already described, the face model deforming unit 92 selects one processing target node from the nodes constituting the face model 330. In FIG. 16, it is assumed that the node 410 is selected as a processing target node. Assume that virtual markers 412A,..., 412E exist in the vicinity of the node 410. The face model deformation unit 92 calculates the distance between the node 410 and the virtual marker based on the coordinates of the node 410 and the coordinates of the virtual marker virtual markers 412A,. Then, the virtual marker 412A located closest to the node 410 is selected from the virtual markers.

  Subsequently, the face model deformation unit 92 checks whether or not the line segments 414A,..., 414E connecting the node 410 and the virtual markers 412A,. That is, first, it is checked whether or not the line segment 414A connecting the node 410 and the virtual marker 412A crosses the boundary edge 400. In the example shown in FIG. 16, the line segment 414A does not cross the boundary edge 400. Therefore, the face model deforming unit 92 sets the virtual marker 412A as one of the corresponding markers of the node 410. The virtual marker 412B next to the virtual marker 412A and next to the node 410 is selected from the virtual markers, and the inspection is performed. A line segment 414B connecting the node 410 and the virtual marker 412B crosses the boundary edge 400. Therefore, the virtual marker 412B is excluded from the corresponding marker of the node 410.

  The face model deforming unit 92 repeats the above operation until a predetermined number (three) of corresponding markers are selected, and the corresponding markers of the node 410 (virtual markers 412A, 412D, and 412E in the example shown in FIG. 16). Select.

  Referring to FIG. 14 again, the face model deforming unit 92 moves the feature points in the triple visual element unit of a certain frame based on the correspondence relationship between the feature points and the virtual markers stored in the face model storage unit 90. Each vector is assigned to the corresponding virtual marker 332. Further, the face model deforming unit 92 assigns to each node of the face model 330 a change amount vector v calculated by a predetermined interpolation formula from the change amount indicated by the motion vector of the corresponding virtual marker 332, so that the face model 330 is deformed. The face model deformation unit 92 outputs the deformed face model 330 as a shape model in the frame.

  Of the basic face model 330, the coordinate vector of a certain node is N, and in the basic face model 330, the coordinate of the i-th virtual marker corresponding to the node is Mi (1 ≦ i ≦ 3), When the coordinate of the corresponding marker in the deformed face model is M′i, the face model deforming unit 92 calculates a change vector v of the coordinate of this node by the following interpolation formula. Note that M′i−Mi corresponds to a motion vector of feature points.

レンダリング部９６は、ポリゴンにより表された形状モデルに対するレンダリングを行なうことができるものであればよく、市販のレンダリングエンジンを用いることもできる。アニメーションのフレームレートにしたがい、１フレームごとの顔モデルを上記式にしたがって生成し、レンダリングを行なうことにより、このレンダリングによりえられた画像のシーケンスとしてアニメーションが得られる。アニメーションはアニメーション記憶部９８に記憶される。

The rendering unit 96 only needs to be capable of rendering the shape model represented by the polygon, and a commercially available rendering engine can also be used. In accordance with the frame rate of the animation, a face model for each frame is generated according to the above equation, and rendering is performed, whereby an animation is obtained as a sequence of images obtained by this rendering. The animation is stored in the animation storage unit 98.

  The animation reading unit 100 has a function of reading an animation from the animation storage unit 98 for each frame and converting it into an image, and writing it into the frame memory of the display unit 52 at every frame interval.

[Operation]
The face animation creation system 40 according to the present embodiment operates as follows. The operation of the lip sync animation creating apparatus 40 can be roughly divided into four phases. The first phase is a phase for creating the speech-visual element corpus storage unit 62. The second phase is a phase in which the animation data storage unit 46 is created from the audio data 42 input using the audio-visual element corpus storage unit 62. The third phase is a phase in which an animation is created from the animation data storage unit 46 using the face model and stored in the animation storage unit 98. The last phase is a phase in which the animation stored in the animation storage unit 98 is displayed on the display unit 52. Hereinafter, the operation of the lip sync animation creating apparatus 40 in each phase will be described.

<First Phase: Creation of Speech-Visual Elementary Corpus Storage Unit 62>
Hereinafter, an operation in which the recording system 60 performs recording and generates the speech-visual element corpus storage unit 62 will be described. With reference to FIG.2 and FIG.3, the marker is previously mounted | worn at each feature point 160 of the head 110 of the speaker 50. FIG. In that state, the speaker speaks. In order to enhance the speech / visual element corpus, or in order to include each phoneme in a well-balanced manner, the content of the utterance is determined in advance, and the utterer 50 is uttered with the content. get. The content of this utterance is stored in the utterance text storage unit 204 shown in FIG.

  When the recording is started and the speaker 50 speaks, the recording / recording system 112 records the voice and the moving image of the face at the time of the utterance, and generates the voice / moving picture data 116. The audio / video data 116 is stored in the audio / video storage unit 140. At this time, the camcorder 132 supplies the time code 134 to the MoCap system 114 and assigns the same time code as the time code 134 to the audio / video data 116.

  At the same time, the position of the feature point 160 at the time of utterance is measured as three-dimensional data by the MoCap system 114 as follows. Each marker moves following the movement of the corresponding feature point. Each of the infrared cameras 136 </ b> A,..., 136 </ b> F captures infrared reflected light from the marker at a predetermined frame rate (for example, 120 frames per second) and outputs the video signal to the data processing device 138. The data processing device 138 assigns the time code 134 to each frame of the video signal, and calculates the three-dimensional coordinates of each marker for each frame based on the video signal. The data processing device 138 collects the three-dimensional coordinates of each marker for each frame and accumulates them as MoCap data 118.

  The audio / moving image data 116 and the MoCap data 118 recorded by the above recording process are given to the data set creation device 122. The data set creation device 122 accumulates the audio / video data 116 in the audio / video storage unit 140 and accumulates the MoCap data 118 in the MoCap data storage unit 142.

  The normalization processing unit 146 reads out the MoCap data in the frame at t = 0 from the MoCap data storage unit 142. The MoCap data at the fixed point at this time becomes a reference for the subsequent normalization process. The normalization processing unit 146 further normalizes the coordinates of each feature point in each frame using the three-dimensional coordinates of a plurality of feature points designated as fixed points as follows.

  That is, in each frame of the MoCap data 152, the normalization processing unit 146 uses the three-dimensional coordinates of the fixed point of the frame and the three-dimensional coordinates of the fixed point in the frame at t = 0 to obtain the equation (1). An affine matrix is obtained, and the three-dimensional coordinates of each feature point are affine transformed using the affine matrix. By this conversion, the three-dimensional coordinates of the feature points after the conversion represent the positions of the feature points of the face in the state where the head is fixed at the position at t = 0 and the speech is performed. That is, the three-dimensional coordinates of each feature point are normalized. By subtracting from the coordinates of each feature point when t = 0 from these coordinates, a motion vector at that time point of the feature point is obtained. As a result, a face parameter series 126 is obtained from the MoCap data 152. The face parameter series 126 is provided to the combining unit 148.

  Referring to FIG. 5, the framing processing unit 200 of the triple visual element data string creation unit 144 converts the audio data of the audio / video data 116 stored in the audio / video storage unit 140 into a predetermined frame length and a predetermined frame interval. Frame it and give it to the feature extraction unit 201.

  The feature extraction unit 201 calculates an acoustic feature quantity (MFCC) used in the processing of the Viterbi alignment unit 206 from each frame given from the framing processing unit 200, and gives it to the Viterbi alignment unit 206 as a feature quantity 230. At this time, audio data of each frame is also provided to the Viterbi alignment unit 206.

  The Viterbi alignment unit 206 performs Viterbi alignment for the series of feature quantities 230 using the acoustic model stored in the acoustic model storage unit 202 and the utterance text stored in the utterance text storage unit 204, and obtains an alignment result. The phoneme label string (phoneme label) string is stored in the phoneme data string storage unit 208 as the phoneme data string 232 together with the duration of each phoneme. At this time, audio data of each frame is also attached to the phoneme data string 232.

  The phoneme-visual element conversion unit 212 sequentially reads the phoneme data from the phoneme data string storage unit 208, and converts the phoneme labels included in each phoneme data into the phoneme-visual element conversion table stored in the phoneme-visual element conversion table storage unit 210. Refer to the table and convert it to a visual label. The phoneme data storing the visual element label instead of the phoneme label constitutes visual element data. At this time, when the same visual element label is continuous, the phoneme-visual element conversion unit 212 collects them into one visual element data and totals the continuation length. Furthermore, the average power of the sound is calculated and assigned to each visual element data. The phoneme-visual element conversion unit 212 stores the visual element data string 234 thus obtained in the visual element data string storage unit 214 together with the framed audio data.

  The visual element-triple visual element conversion unit 216 sequentially reads visual element data from the visual element data string storage unit 214 and performs the following processing. That is, the visual element-triple visual element conversion unit 216 sets the visual element label of each visual element data, the visual element label included in the immediately preceding visual element data, the visual element label of the visual element data, and the immediately following element. The visual element labels are converted into triple visual element labels combined in this order. Visual element data in which triple visual element labels are stored instead of visual elementary labels in this way is triple visual element data. The visual element-triple visual element conversion unit 216 stores the triple visual element data string 236 thus obtained in the triple visual element data string storage unit 218 together with the audio data.

  The combining unit 148 uses the time information attached to the triple visual element data sequence stored in the triple visual element data sequence storage unit 218 and the face parameter series 126 given from the normalization processing unit 146. The audio-visual element corpus is generated together with the audio data 42 by being synchronized and combined and stored in the audio-visual element corpus storage unit 62.

<Second Phase: Synthesis of Animation Data Storage Unit 46>
The second phase is performed by the animation data synthesizer 44. Voice data 42 representing the voice of the character is prepared and provided to the triple visual element data string creation unit 80. The voice data 42 is obtained by recording in advance what is spoken by a speaker (or voice actor) who is in charge of the voice of the character. The speech text of the speech data 42 is stored in the speech text storage unit 286 shown in FIG.

  Referring to FIG. 10, framing processing section 280 frames input audio data 42 with the same frame length and frame interval as framing processing section 200 shown in FIG.

  The feature quantity extraction unit 282 extracts a predetermined acoustic feature quantity (MFCC) for each frame of the voice by the same processing as the feature extraction unit 201 shown in FIG. 5 and gives it to the Viterbi alignment unit 288.

  The Viterbi alignment unit 288 performs Viterbi alignment for the feature amount extraction unit 282 using the acoustic model storage unit 284 and the utterance text storage unit 286, and generates a phoneme data string including phoneme labels and phoneme data including the duration of each phoneme. The phoneme-visual element conversion unit 292 is provided.

  The phoneme-visual element conversion unit 292 refers to the phoneme-visual element conversion table stored in the phoneme-visual element conversion table storage unit 290 for the phoneme label in each phoneme data included in the phoneme data string. And convert it to a visual label. The phoneme data in which the visual element label is stored instead of the phoneme label becomes visual element data, which is output from the phoneme-visual element conversion unit 292 as a visual element data string and stored in the visual element data string storage unit 293. Each visual element data includes a visual element label and a duration of the original phoneme. If the same visual element labels are consecutive, they are put together and the duration is also summed. In addition, the average power of the corresponding sound is calculated for each visual element data.

  The visual element-triple visual element conversion unit 294 sequentially reads each visual element data stored in the visual element data string storage unit 293, and the visual element labels included in each visual element data are displayed immediately before and after the visual element label. The visual element label of the visual element data is replaced with the triple visual element label obtained by combining with the visual element label of the data, and is stored in the triple visual element data string storage unit 295 as a triple visual element data string.

  Referring to FIG. 1, triple visual element unit selection unit 82 reads triple visual element data sequence 296 from triple visual element data sequence storage unit 295 shown in FIG. 10, and performs the following processing. That is, the triple visual element unit selection unit 82 performs the triple visual element having the same triple visual element label included in the speech-visual element corpus storage unit 62 for each triple visual element data included in the triple visual element data string 296. A unit is searched (step 314 in FIG. 13). If there is such a unit, the cost is calculated by using the visual element continuation length and the average power included in the triplet visual element data among all such units (step in FIG. 13). 318). If there is no such unit, the first half pair visual element label or the second half pair visual element label of the triple pair visual element labels is matched with the voice-visual element corpus storage unit 62. The cost is calculated according to the above-described equation (2) using the visual element continuation length and the average power included in the triple visual element data among the read data and all of them (step 316 in FIG. 13).

  Then, the triple visual element unit that gives the minimum cost calculated in this way is selected as the optimal triple visual element unit for the triple visual element data to be processed (step 322 in FIG. 13).

  The triple visual element unit selection unit 82 gives the triple visual element unit connection unit 84 a triple visual element unit sequence including the triple visual element units obtained in this way.

  The triple visual element unit linking unit 84 sets the motion vector sequence of each triple visual element unit in the triple visual element unit sequence given from the triple visual element unit selection unit 82 as the motion vector sequence of the preceding triple visual element unit. , And the subsequent motion vector sequence of the triplet visual element unit on the time axis. At this time, as described with reference to FIG. 17, the last part of the motion vector sequence of each unit is extended by time T, and between the portion of the motion vector sequence of the succeeding unit at the beginning of time T. Then, the smoothing process according to the above equation (3) is performed for each feature point.

  Through the above processing, animation data is created. The created animation data is stored in the animation data storage unit 46.

<Third phase: Animation using model>
The creation of the animation is performed by the animation creation device 48 shown in FIG. With reference to FIG. 1, the face model deforming unit 92 reads the face model stored in the face model storage unit 90. With respect to this face model, the feature points when the speech-visual element corpus storage unit 62 is created are already associated with the virtual markers, and the virtual markers corresponding to the respective nodes constituting the face model have already been established. It shall be stipulated.

  The face model deforming unit 92 sequentially reads out the animation data of the frame having the time closest to the time according to the animation frame rate from the animation data stored in the animation data storage unit 46. Perform the following process.

  The face model deforming unit 92 assigns a three-dimensional motion vector of a corresponding feature point included in the read animation data to each virtual marker of the face model. The face model deforming unit 92 further calculates a variation vector v that gives the three-dimensional position coordinates of each node in the face model storage unit 90 according to the equation (4). By calculating the variation vector for all nodes, the face model in the frame is completed. This face model is given to the rendering unit 96.

  The rendering unit 96 renders the face model given from the face model deforming unit 92 according to the rendering data stored in the rendering data storage unit 94 and the settings, and creates an image corresponding to one frame of animation. And stored in the animation storage unit 98.

  The above operations of the face model deforming unit 92 and the rendering unit 96 are repeated, and when the end of the animation data stored in the animation data storage unit 46 is reached, the animation creating device 48 ends the process.

  With the above processing, the animation storage unit 98 stores a series of face images at a predetermined frame rate representing the face animation corresponding to the input audio data 42.

<Fourth phase: Animation display>
The animation is displayed by the animation reading unit 100 and the display unit 52. In the display process, the animation reading unit 100 sequentially reads the images stored in the animation storage unit 98 from the top, and writes them in a frame memory (not shown) in the display unit 52 at a predetermined time interval. The display unit 52 reads the images written in the frame memory at predetermined time intervals and displays them on the screen. As a result, on the screen of the display unit 52, the face animation defined by the face model of the animation character stored in the face model storage unit 90, which changes corresponding to the input voice data 42, is displayed. .

  As described above, according to the lip sync animation creating apparatus 40 according to the present embodiment, a large number of feature points of the speaker's face are associated with each node of the face model 170 in advance. Furthermore, the phoneme label obtained from the speech at the time of utterance is converted into a corresponding visual element label, further converted into a triple visual element label, and the duration, the average power of the voice during the duration, and motion capture A voice-visual element corpus is created by combining the obtained three-dimensional motion vector sequence of facial feature points at the time of utterance and storing it in the voice-visual element corpus storage unit 62 as a triple visual element unit.

  When the audio data 42 is given, a phoneme data string composed of the phoneme label obtained from the audio data 42, the phoneme duration, and the average power is created, and each phoneme label is also the same as when the speech-visual element corpus is created. A triplet visual element data string is obtained by converting the triplet visual element label by the method. Based on the continuation length of each triplet visual element label and the average power of the voice, a triplet visual element unit having a minimum cost is selected from the voice-visual element corpus storage unit 62 using an evaluation function of a predetermined cost. These motion vector sequences are connected by a triple visual element unit connecting unit 84. In this connection, among the motion vector sequences of adjacent triple visual element units, the motion vector sequence of the preceding triple visual element unit is extended and smoothed with the motion vector of the subsequent triple visual element unit in this part. The process is performed.

  Through such processing, a motion vector representing the trajectory of each node of the face model 170 is calculated based on the motion vector of the facial feature point at the time of actual speech stored in the speech-visual element corpus storage unit 62. Therefore, the temporal change of the face model as a set of nodes can be obtained as animation data representing a natural movement close to the actual movement. By calculating the motion vector of each node of the face model 170 based on the motion vector of each feature point constituting the animation data and the corresponding relationship with each node of the face model 170, a set of each node is obtained for each frame. Face models can be created, and a series of deformed face models is obtained. An animation can be created by rendering these face models.

  The face parameter series 126 stored in the animation data storage unit 46 is obtained by actually capturing a nonlinear trajectory of each feature point of the face when the voice represented by the voice data 42 is spoken. Express based on measured data. Therefore, it is possible to create a natural animation that faithfully reproduces non-linear changes in facial expressions during speech.

  The lip sync animation creation device 40 creates an animation on a model basis. After the voice-visual element corpus storage unit 62 is created, the user performs the Viterbi for the voice data 42 corresponding to the voice of the character, the face model defining the shape of the character's face in a stationary state, and the voice data 42. An acoustic model for alignment and utterance text corresponding to the voice data 42 are prepared, and the facial expression changes according to the voice of the character simply by designating a virtual marker corresponding to the feature point on the face model. Create natural lip-sync animations. The shape of the character's face represented by the face model 44 may be arbitrary without limiting the design of the character's face. Therefore, a lip sync animation can be created without narrowing the variation of animation production by the user.

  In the above-described embodiment, the lip sync animation creating apparatus 40 includes all of the recording system 60, the animation data synthesizing apparatus 44, the animation creating apparatus 48, and the animation reading unit 100. However, the present invention is not limited to such an embodiment. These may all be realized by separate apparatuses or programs. Further, it is not necessary that these are physically implemented on the same computer, and it is not necessary that each unit constituting the animation data synthesizing apparatus 44 is implemented on a single computer. These may be realized by a program operating on different computers, and data movement between them may be realized via a network or via a removable recording medium.

[Realization and operation by computer]
Each functional unit of the lip sync animation creating apparatus 40 according to the present embodiment, except for some special devices included in the recording / recording system 112 and the MoCap system 114 of the recording system 60 (see FIGS. 1 and 2), Both are realized by computer hardware, a program executed by the computer hardware, and data stored in the computer hardware. FIG. 18 shows the external appearance of the computer system 450, and FIG. 19 shows the internal configuration of the computer system 450.

  Referring to FIG. 18, a computer system 450 includes a computer 460 having a DVD (Digital Versatile Disk) drive 472 and a memory port 470 in which a removable memory can be mounted, a keyboard 466, a mouse 468, a monitor 462, A microphone 490 and a pair of speakers 458 are included. The microphone 490 is used when recording the audio data 42 (see FIG. 1) in the computer system 450. The speaker 458 is used to reproduce sound when displaying an animation.

  Referring to FIG. 19, in addition to memory port 472 and DVD drive 470, computer 460 is connected to hard disk 474, CPU (Central Processing Unit) 476, CPU 476, hard disk 474, memory port 472, and DVD drive 470. Bus 486, a read only memory (ROM) 478 for storing a boot-up program and the like, a random access memory (RAM) 480 connected to the bus 486 and storing a program command, a system program, work data, and the like, A sound board 488 connected to the bus 486 and used to convert the audio signal from the microphone 490 into a digital signal and to convert the digital audio signal output from the CPU 476 into an analog signal to drive the speaker 458. The computer system 450 may further include a printer.

  Computer 460 further includes a network interface (I / F) 496 that provides a connection to a local area network (LAN) 452.

  A computer program for causing the computer system 450 to realize each function unit of the lip sync animation creating apparatus 40 is stored in the DVD 482 or the memory 484 inserted into the DVD drive 470 or the memory port 472 and further transferred to the hard disk 474. Alternatively, the program may be transmitted to the computer 460 through a network (not shown) and stored in the hard disk 474. The program is loaded into the RAM 480 when executed. The program may be loaded into the RAM 480 directly from the DVD 482, from the memory 484, or via a network.

  This program includes a plurality of instructions for causing the computer 460 to realize each functional unit of the lip sync animation creating apparatus 40 of this embodiment. Some of the basic functions necessary to realize this function are provided by modules of various toolkits installed in the computer 460, or an operating system (OS) or a third-party program running on the computer 460. . Therefore, this program does not necessarily include all functions necessary for realizing the system and method of this embodiment. This program executes processing performed by each functional unit of the above-described face animation creation system 40 by calling an appropriate function or “tool” in a controlled manner so as to obtain a desired result. It is only necessary to include instructions to be executed. The operation of computer system 450 is well known and will not be repeated here.

[Various variations]
In the above-described embodiment, triple visual element labels are used as visual element labels of visual element corpus and visual element data. By doing so, it is possible to select an appropriate visual element unit in a form that reflects the movement of the face in the utterance before and after the utterance corresponding to a certain visual element. However, the present invention is not limited to such an embodiment. For example, only one visual label may be used. In this case, although the discontinuous part of the obtained visual element unit may become large, it can connect smoothly by the above-mentioned weighted addition process.

  In the above-described embodiment, as the triple visual element label, one visual element label around the certain visual element is adopted one by one and combined with the central visual element label. However, the present invention is not limited to such an embodiment. For example, a pair of visual element labels including a visual element label of a visual element before or after a certain visual element and a visual element label of a target visual element may be used. Moreover, you may employ | adopt what consists of four or more visual element labels. When a set of four or more visual element labels is adopted, there is a high possibility that a visual element unit having an appropriate visual element label group is not found in the visual element corpus. In such a case, the visual element corpus may be made larger, or an alternative visual element unit may be searched using a smaller number of sets of visual element labels as in the above-described embodiment. .

  In the above-described embodiment, only the motion vector of the preceding visual element unit is expanded by time T when the visual element units are connected. However, the present invention is not limited to such an embodiment. For example, any visual element unit may extend the motion vector by T / 2 before and after it. Also, the extension time may be set to a value deemed appropriate by a subjective test.

  In the above-described embodiment, in order to determine which visual element unit should be selected, the utterance continuation length of the phoneme corresponding to the visual element and the average power of the sound during that time are used. However, the present invention is not limited to such an embodiment. Other prosodic features such as pitch (fundamental frequency) may be used, or any combination thereof may be used. Due to the characteristics of the animation that the face image changes with time, it is desirable to adopt the utterance duration as an object of evaluation, but after selecting a visual unit using prosodic features other than the utterance duration, utterance continuation The length may be adjusted according to the duration of the visual element data.

  Further, in the above-described embodiment, a triple visual element unit having prosodic features is searched by Equation (2), and such a unit is handled as suitable for animation synthesis. However, the formula that can be used is not limited to formula (2). The sum of squares of differences in prosodic features may be used instead of equation (2), and other equations may accurately represent the prosodic feature differences between visual element units and visual element data. You can use such a thing.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

It is a block diagram of the lip sync animation creating apparatus 40 according to the embodiment of the present invention. 3 is a block diagram showing a detailed configuration of a recording system 60. FIG. It is a figure which shows the example of arrangement | positioning of the marker with which the head 110 is mounted | worn. It is a schematic diagram showing a procedure for creating an animation 172 of a face image from a face model 170 of an animation character and a motion vector for each frame. It is a block diagram of the triple visual element data string creation unit 144. It is a schematic diagram which shows the outline of Viterbi alignment. It is a figure which shows the structure of the triple visual element data sequence 234. It is a figure which shows the structure of a phoneme-visual element conversion table. It is a figure which shows the structure of the audio-visual element corpus memory | storage part 62. FIG. It is a figure which shows the more detailed structure of the triple visual element data sequence production | generation part 80. FIG. It is a figure which shows the structure of the triple visual element data sequence 296. FIG. It is a figure which shows the structure of the triple visual element data row | line | column 300. FIG. It is a flowchart which shows the control structure of the computer program which implement | achieves the triple visual element unit selection part 82. FIG. It is a schematic diagram which shows a face model. It is a flowchart which shows the control structure of the computer program which implement | achieves the marker labeling process performed by the face model deformation | transformation part 92. FIG. It is a figure for demonstrating matching with the node around a lip in a face model, and a virtual marker performed by the face model deformation | transformation part 92. FIG. It is a figure for demonstrating the connection method of the motion vector by the triple visual element unit connection part 84. FIG. It is a figure which shows an example of the external appearance of the computer system which implement | achieves the main functions of the lip-sync animation production apparatus 40 concerning one embodiment of this invention. It is a block diagram of the computer system shown in FIG.

Explanation of symbols

40 Lip Sync Animation Creation Device 44 Animation Data Synthesizer 48 Animation Creation Device 60 Recording System 62 Audio-Visual Element Corpus Storage Unit 80 Triplet Visual Element Data Sequence Creation Unit 82 Triplet Visual Element Unit Selection Unit 84 Triplet Visual Element Unit Connection Unit 92 Face Model transformation unit 96 Rendering unit 100 Animation reading unit 112 Recording / recording system 114 MoCap system 116 Audio / moving image data 122 Data set creation device 138 Data processing device 144 Triple visual element data string creation unit 146 Normalization processing unit 148 Coupling unit 200, 280 Framing processing unit 201,282 Feature extraction unit 202,284 Acoustic model storage unit 204,286 Speech text storage unit 206,288 Viterbi alignment unit 210,29 Phoneme-visual element conversion table storage unit 212,292 Phoneme-visual element conversion unit 214,293 Visual element data string storage unit 216,294 Visual element-triple visual element conversion unit 218,295 Triple element visual element data string storage unit 234,296 Triple visual element data string 240 Speech waveform data 242 Motion vector string 244 Triple visual element unit string 300 Triple visual element data string

Claims (7)

An animation data creation program for creating animation data of a face including a mouth that moves in accordance with audio data, based on the input audio data, in a computer comprising first storage means for storing a visual element corpus There,
The visual element corpus includes a plurality of visual element units created from a face image during speech with sound,
Each visual element unit is a continuation of the phoneme corresponding to the visual element unit obtained from the visual element label, the motion data indicating the movement of the face corresponding to the visual element unit, and the sound corresponding to the visual element unit. Including prosodic information including length,
The program causes the computer to function as first conversion means for converting the voice data into a phoneme data string that identifies a phoneme represented by the voice data,
The phoneme data string includes phoneme data including a phoneme label and prosodic information including a duration of the phoneme portion in the speech data;
The program further outputs a visual element data string by converting each of the phoneme labels included in the phoneme data in the phoneme data string output by the first conversion means into a corresponding visual element label. Causing the computer to function as the second conversion means,
The visual element data string output from the second conversion means is a visual element label and a continuation length of a phoneme corresponding to at least the visual element data obtained from a portion corresponding to the visual element data in the audio data. Visual element data consisting of prosodic information including
The program further includes
Each of the visual element data included in the visual element data string has the same visual element label as the visual element label included in the visual element data in the visual element unit in the visual element corpus, and the visual element data An evaluation function that evaluates the similarity of speech based on the prosodic information included in the visual element corpus and the prosodic information included in each visual element included in the visual element corpus, and having the speech most similar to the speech included in the visual prime data First selecting means for selecting an evaluated visual element unit from the visual element corpus;
By connecting the motion data included in the visual element unit selected by the first selection means on the time axis according to the order of the visual element data string, the animation data of the mouth corresponding to the input audio data is obtained. An animation data creating program for causing the computer to function as a connecting means for creating. The prosodic information of speech included in each of the visual element units included in the visual element corpus includes the average power of the speech during the duration in addition to the duration of the speech corresponding to the visual elementary unit,
The first conversion means includes means for converting the voice data into a phoneme data string, and the phoneme data string includes a phoneme label, a continuation length and an average power of the phoneme part in the voice data. Phoneme data consisting of
The first selection means includes
For each visual element data included in the visual element data string, among visual element units in the visual element corpus, each visual element unit having the same visual element label as the visual element label included in the visual element data. An evaluation means for evaluating a value of an evaluation function for evaluating the similarity of speech based on the duration and average power included in the visual element data and the duration and average power of the visual element unit;
The second evaluation means for selecting from the visual element corpus the visual element unit evaluated by the evaluation means as having the sound most similar to the sound included in the visual element data. The animation data creation program described. The computer further includes second storage means for storing a phoneme-visual element conversion table storing a correspondence relationship between a phoneme label and a visual element label,
The second conversion means refers to each of the phoneme labels included in the phoneme data of the phoneme data string output from the first conversion means by referring to the phoneme-visual element conversion table, thereby corresponding visual element labels. The animation data creation program according to claim 1, further comprising means for converting into a visual element data string. Each visual element unit of the visual element corpus includes a first number of visual element units preceding each visual element when the plurality of visual element units are created from the face image at the time of speech with speech. A visual element label, and a visual element label of a second number of visual element units following each visual element, the visual element label of the preceding first number of visual element units, and each visual element A visual element label of the unit and a visual element label of the second number of visual element units subsequent to each other constitute a set of visual element labels;
The second conversion means includes
For each of the phoneme data in the phoneme data string output by the first conversion means, a phoneme label included in the phoneme data, and a phoneme label included in the preceding first phoneme data; Means for converting each of the phoneme labels included in the second number of phoneme data thereafter to a corresponding visual element label and combining them in the order of the phoneme data to create a set of visual element labels;
For each of the phoneme data in the phoneme data string output by the first conversion means, the phoneme label included in the phoneme data in the phoneme data string output by the first conversion means is the visual element label. Means for generating and outputting the visual element label data by replacing with a set of visual element labels obtained by the means for generating
The first selection means includes, for each of the visual element data included in the visual element data string, the same visual element label as the set of visual element labels included in the visual element data to be processed in the visual element corpus. The visual function is evaluated by an evaluation function that evaluates the similarity of speech using prosodic information included in the visual element data to be processed and prosodic information included in each visual element unit included in the visual element corpus. The animation data creation program according to claim 1, further comprising: second selection means for selecting from the visual element corpus a visual element unit evaluated as having the sound most similar to the sound included in the elementary data. The second selection means includes
For each of the visual element data included in the visual element data string, a visual element unit having the same set of visual element labels as the set of visual element labels included in the visual element data to be processed is included in the visual element corpus. A determination means for determining whether or not it exists;
In response to the determination by the determination means that a visual element unit having the same set of visual element labels as the set of visual element labels included in the visual element data to be processed exists in the visual element corpus. For each of these visual element units, a value of an evaluation function for evaluating the similarity of speech is calculated from the prosodic information included in the visual element data and the prosodic information included in each visual element unit included in the visual element corpus. First calculating means for performing,
In response to determining by the determination means that there is no visual element unit having the same set of visual element labels as the set of visual element labels included in the visual element data in the visual element corpus. From the set of visual elementary labels of the target visual elementary data, only a partial visual elementary label consisting of a part including the visual elementary label of the visual elementary data to be processed is used as a reference from within the visual elementary corpus. Means for selecting a visual element unit having a set of visual element labels whose positions and contents coincide with each other;
Second calculating means for calculating a value of the evaluation function with respect to each of the visual element units selected by the selecting means with respect to the prosodic information included in the visual element data to be processed;
The animation data creation program according to claim 4, further comprising means for selecting a visual element unit having the smallest value of the evaluation function calculated by the first calculation means or the second calculation means. The connecting means includes, for the motion data of two visual element units continuous on the time axis among the motion data included in the visual element unit selected by the selection means, the last of the motion data of the preceding visual element unit. To link the motion data of the visual element unit on the time axis by adding each of the motion data of the partial part and the motion data of the first part of the subsequent visual element unit with weighting according to time. The animation data creation program according to claim 1, comprising weighted addition means. An animation data creation device for creating animation data indicating a movement of a face corresponding to input audio data using a visual element corpus including a plurality of triple visual element units,
Each of the triple visual element units includes a triple visual element label, a visual element duration corresponding to the triple visual element unit, an average power of speech spoken when the visual element is recorded, and the visual element. Including movement data of feature points of the speaker's face when the element was recorded,
Phoneme conversion means for creating a phoneme data string comprising a phoneme label, a phoneme duration, and an average power at the time of speech of the phoneme by performing speech analysis on the input speech data;
Means for storing a table indicating the correspondence between phoneme labels and visual element labels;
A first conversion means for creating a visual element data string by converting a phoneme label included in the phoneme data string into a corresponding visual element label with reference to the table;
For each of the visual element data in the visual element data string output by the first conversion means, the visual element label is converted into a triple visual element label combined with the visual element labels of the preceding and subsequent visual element data, and a triple visual element is obtained. Second conversion means for outputting a data string;
For each of the triple visual element data included in the triple visual element data string output from the second conversion means, a triple visual element label matching the triple visual element label of the triple visual element data is obtained from the visual element corpus. A triplet visual element unit, wherein the triplet visual element unit is evaluated by an evaluation function for evaluating a similarity between the duration and power of the triplet visual element unit and the duration and average power of the triplet visual element data. A selection means for selecting a triple visual elementary unit evaluated to have a duration and average power similar to the duration and average power of the elementary data;
The face motion data included in the triple visual element unit selected by the triple visual element unit selection unit is connected on the time axis according to the time series of the triple visual element data, thereby creating face animation data. And an animation data creating device.

Images