JP2007034788A - Head motion learning device and head motion composition device for head motion automatic generation, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a head motion learning device and a head motion composition device for head motion automatic generation capable of reflecting sensitivity of a user. <P>SOLUTION: This head motion learning device includes: a feeling intensity input part 64 for receiving input of a feeling intensity parameter showing intensity related to prescribed feeling of an utterance subject during utterance from the user; and a learning part 78 for learning relation between motion of a head of the utterance subject, and a prescribed acoustic characteristic amount and the feeling intensity parameter from information showing the motion of the head of the utterance subject during the utterance and the feeling intensity parameter input through the feeling intensity input part 64, related to the prescribed acoustic characteristic amount extracted as a time series from voice of the utterance subject during each utterance, and the utterance. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an apparatus that automatically generates a head motion of a synthesized face image in accordance with a voice, and in particular, a head that allows a user to freely customize the intensity of an emotion desired to be expressed by the generated head motion. The present invention relates to a head motion learning device, a head motion synthesis device, and a computer program therefor.

  In recent years, production of animations using characters created by CG has become popular. Therefore, the demand for such animation production technology is increasing year by year, and the advancement of the technology is also highly expected.

  In the production of animation, it is necessary to create an image synchronized with the sound. Usually, an animation image is created first, and then sound is blown in accordance with the image. However, in order to create an animation image first, imagination that matches the progress of the story is required. In addition, in order to inject sound in accordance with an image, a technique for in charge of sound is required. On the other hand, there is a method in which a character's voice is first blown in accordance with a script and an animation image of the character is automatically generated based on the voice. In this case, since the image is automatically generated from the sound, the sound and the image are naturally synchronized, and a high-quality animation can be produced. Also, since it is not necessary to create an image in accordance with the progress of the story while predicting the voice of the character, it is possible to efficiently produce the animation.

  Character movement is one of the important image movements that are problematic in animation. This character's movement includes facial expression, gesture, head movement, and the like. These are important elements that can make it easier for those who see the image to understand the emotions they want to express with the character. As described above, when using the motion of an image to facilitate understanding of emotion, the motion needs to be sufficient to help understand the emotion. If so, it is desirable that the emotions to be expressed by the images can be naturally understood by those who have seen the movement of the character.

  As described above, the character motion includes facial expressions, gestures, head motions, and the like. Of these, facial expressions, gestures, etc. have a relatively large individual difference in actions corresponding to certain emotions, and also greatly depend on utterance intentions. Therefore, it is difficult to automatically generate facial expressions, gestures, etc. that can naturally guess the emotions of individual characters from a certain voice, even if anyone sees it. That is, a certain facial expression or gesture is not necessarily applicable to all characters. On the other hand, as for the head movement, the same movement can be applied to other characters to some extent without a sense of incompatibility compared to other movements. Therefore, when automatically generating a character motion from speech, it is desirable to adopt a method of generating a head motion so that a person who has seen the image can properly understand the emotion of the character.

  Thus, when automatically generating a natural head movement from speech, a method of using parameters learned in advance can be considered. In this learning, a method is adopted in which voices and actions are collected at the same time and the data are accumulated to learn the relationship between what kind of actions are appropriate from what kind of voices. As a method for generating a head movement from speech using a relationship obtained from such learning, there is a method using a neural network as shown in Non-Patent Document 1. A neural network is a non-linear method and can generate movements that closely resemble actual human head movements.

  FIG. 1 shows an automatic head movement generation system according to the prior art disclosed in Non-Patent Document 1. Referring to FIG. 1, the head movement automatic generation system captures a head movement 30 of a speaker for learning and collects head movement data for learning, and an utterance. A microphone 34 for recording a person's utterance voice 32, a voice storage unit 38 for storing voice for learning recorded by the microphone 34, and data relating to head movements collected by the camera 36 are stored. A head motion storage unit 40. Time information is recorded in both the recorded voice and the data related to the head movement. This time information is shared between the recording device and the recording device, and therefore it is possible to take a correspondence relationship between the sound and the head image.

  The head movement automatic generation system further calculates an acoustic feature quantity from the voice sharing the time, and from this feature quantity data and the data relating to the head movement, learning data for neural network learning (this is referred to as “voice-head. Voice-head motion synchronization unit 42 for creating the voice-head motion data for storing the voice-head motion data created by the voice-head motion synchronization unit 42. A data storage unit 44, a learning unit 46 for causing a predetermined neural network to learn about a relationship between a voice and a head motion synchronized with the data, using the stored voice-head motion data, and the learning And a neural network parameter storage unit 48 for storing the parameters of the neural network obtained by the above.

  The head movement automatic generation system further includes a voice storage unit 52 for storing voice used for the character voice 54, which is recorded in advance according to the script, and a neural network parameter stored in the neural network parameter storage unit 48. And a head movement generation unit 50 for generating a head movement 56 of the character based on the voice stored in the voice storage unit 52.

  In this head movement automatic generation system, a speech voice 32 of a speaker is recorded by a microphone 34 and stored in a voice storage unit 38. On the other hand, the head motion data of the speaker recorded by the camera 36 is stored in the head motion storage unit 40. Both include common time information.

  The voice-head movement synchronization unit 42 uses the data that shares time information among the voice stored in the voice storage unit 38 and the data related to the head movement stored in the head movement storage unit 40, to determine a predetermined time. Speech-head motion data for learning the neural network is generated. The generated speech-head motion data is stored in the speech-head motion data storage unit 44. Using this speech-head motion data, the learning unit 46 causes the neural network to learn the relationship between the speech and the corresponding head motion. Parameters (neural network parameters) that define the operation of the neural network obtained by the learning are stored in the neural network parameter storage unit 48.

  Using this neural network parameter, the head motion generation unit 50 automatically generates head motion from speech. Specifically, first, the voice of the character is recorded based on the script and stored in the voice storage unit 52. On the other hand, the neural network is set in advance by the parameters stored in the neural network parameter storage unit 48. From the voice stored in the voice storage unit 52, the same kind of acoustic feature quantity as that calculated by the voice-head movement synchronization unit 42 at the time of learning is calculated. By giving this acoustic feature quantity to the neural network as an input, the movement of the head (coordinates of the head feature points) corresponding to the input speech is output from the neural network as the output. By calculating this value for each frame, the head action 56 of the character is automatically generated.

In this way, the voice of the character is first recorded based on the script, and then the head movement synchronized with the voice is generated. Since this head movement is generated by a neural network based on learning, it is natural. Therefore, it is easy for the person who sees the image to understand the emotion of the character represented by the head movement, and for the user, there is an advantage that the animation can be efficiently produced.
Shinichi Kawamoto, Yoshinori Matsushita, Mitsuru Nakai, Hiroshi Shimohira, Shigeki Hiyama, “Automatic Generation of Head Behavior during Speech for anthropomorphic Spoken Dialogue Agent”, Journal of the Acoustical Society of Japan, Autumn 2002.

  As in the technique disclosed in Non-Patent Document 1, when the head movement is generated automatically from the acoustic feature amount calculated from the voice, a natural head movement synchronized with the voice is generated in terms of the efficiency of image generation. There is no problem with that it can. However, if the head movement is fully automatically generated from the acoustic feature amount calculated from the voice, a head movement that has no room for reflecting the sensitivity of the creator of the animation that is the user is generated. In other words, the head movement of the character is limited by the acoustic features of the voice collected at the time of learning and the voice recorded according to the script, and various character personalities can be expressed by the head movement. It becomes difficult.

  Accordingly, an object of the present invention is to provide a head motion learning device and a head motion synthesis device for automatically generating head motion that can reflect the user's sensibility.

  The head movement learning device according to the first aspect of the present invention includes an emotion strength input means for receiving an input of an emotion strength parameter indicating the strength related to a predetermined emotion of the utterance subject at the time of utterance, and the utterance of each utterance. A predetermined acoustic feature extracted as a time series from the speech of the utterance subject at the time, the emotion strength parameter input via the emotion strength input means regarding the utterance, and the movement of the head of the utterance subject at the time of the utterance And learning means for learning a relationship between a predetermined acoustic feature amount and emotion intensity parameter and head movement of the utterance subject.

  According to this head movement learning device, the user can input an arbitrary emotion intensity parameter in learning the above-described relationship. It is possible to learn the relationship between emotion intensity parameters, acoustic features, and head movements that the user thinks necessary. After learning, an appropriate head motion can be generated based on the relationship described above for an emotion intensity parameter having an arbitrary value. Therefore, it is possible to provide an appropriate head movement learning device for reflecting the user's sensitivity to the head movement learning result.

  Preferably, the learning means receives the voice at the time of the utterance of the utterance subject, and extracts the acoustic feature quantity of the voice every predetermined time from the start of the utterance, and the acoustic feature quantity extraction means Time information providing means for attaching information indicating the time from the start of the utterance to the extracted acoustic feature amount, and information indicating the position or direction of the head of the utterance in association with the time The head position information acquisition means for acquiring the utterance, the acoustic feature quantity extracted by the acoustic feature quantity extraction means for a certain utterance, and the time information from the utterance given by the time information giving means, and the emotion intensity for the utterance Synchronizing means for generating learning data by synchronizing the emotion intensity parameter input by the input means and the position or orientation of the head of the utterance subject acquired by the head position information acquiring means Using the learning data generated by the synchronization means, means for learning the relationship between the time series of the acoustic feature amount and the emotion intensity parameter and the change in the position or orientation of the head of the utterance subject. Including.

  According to this head movement learning device, time information is added to the acoustic feature amount extracted from the voice, and the head position or orientation corresponding to the time from the start of the utterance and the emotion input by the emotion strength input unit The learning data is generated in synchronization with the intensity, and the relationship between the time series of the acoustic feature amount and the emotion intensity parameter and the change in the position or orientation of the head of the utterance subject is learned based on the data. As a result, it is possible to provide an appropriate head motion learning device for learning the relationship between head motions synchronized with speech.

  More preferably, the means for learning uses the learning data generated by the synchronization means, and the time series of the predetermined acoustic feature amount given the time from the start of the utterance of a certain utterance, the emotion intensity parameter, and the utterance subject A function approximation learning means for learning the relationship between the position or orientation of the head of the head by a predetermined nonlinear function approximation.

  According to the head movement learning device, the relationship between the time series of the acoustic feature amount and the emotion intensity parameter and the position or orientation of the head of the utterance subject is learned by nonlinear function approximation. Therefore, it is possible to learn a non-linear change in the position or orientation of the head that cannot be expressed by linear function approximation. As a result, it is possible to provide an appropriate head movement learning device for learning a head movement close to a more natural head movement.

  More preferably, the function approximation learning unit learns the relationship between the acoustic feature amount and the emotion intensity parameter and the position or orientation of the head of the utterance subject using the learning data generated by the synchronization unit as learning data. Including a neural network.

  According to this head movement learning device, the relationship between the time series and emotion intensity parameters of acoustic features and the position or orientation of the head of the utterance subject is learned by nonlinear function approximation using a neural network. Accordingly, it is possible to learn an appropriate head motion for learning a non-linear change in head position or orientation that cannot be learned by linear function approximation without using a neural network. As a result, it is possible to provide an appropriate head movement learning device for learning a head movement close to a more natural head movement.

  When the head motion synthesizer according to the second aspect of the present invention is given a time series of predetermined acoustic features extracted from the speech of the utterance subject at the time of utterance and an emotion intensity parameter specified for the utterance , Head position estimating means for estimating the movement of the head of the utterance when the voice is uttered, acoustic feature quantity extracting means for extracting a time series of predetermined acoustic feature quantities from the voice, and acoustic features By providing the head position estimation means with a time series of predetermined acoustic features extracted by the quantity extraction means and the specified emotion strength parameter, information on the movement of the head of the utterance subject synchronized with the speech is obtained. Head movement generation means for obtaining a series of outputs from the head position estimation means.

  According to this head movement synthesizer, when a voice is input, the head movement of the image is automatically synthesized from the information included in the voice. Therefore, it is possible to provide an appropriate head movement synthesis device for efficiently synthesizing a natural head movement synchronized with the voice.

  Preferably, the head movement synthesizer further includes means for storing the sound in advance and supplying the sound to the acoustic feature amount extraction means.

  According to this head movement synthesizer, a voice can be recorded in advance and stored. Therefore, it is possible to provide an appropriate head motion synthesizer for synthesizing the head motion of an image based on previously recorded sound.

  More preferably, the head motion synthesizer further includes means for providing the head motion generation means with an emotion strength parameter corresponding to the emotion strength input by the user.

  According to this head movement synthesizer, the user can arbitrarily input emotion intensity. Therefore, it is possible to provide a head movement synthesis device that can appropriately reflect the user's sensitivity to the synthesized head movement.

  More preferably, the head position estimating means preliminarily specifies a time series of predetermined acoustic features extracted from the speech at the time of the utterance of the utterance subject, and an emotion intensity designated with respect to the predetermined emotion of the utterance subject at the time of the utterance Based on the parameters and the information indicating the movement of the head of the utterance subject at the time of the utterance, the relationship between the time series of the predetermined acoustic feature amount and the emotion intensity parameter and the position or orientation of the head of the utterance subject is determined in advance. Includes learned machine learning means.

  According to this head motion synthesizer, the machine learning means learns in advance the relationship between the time series of acoustic feature quantities, the emotion intensity parameter, and the information indicating the movement of the head of the utterance subject. Therefore, the user can input the acoustic feature amount and the emotion intensity parameter to the machine learning means, and obtain a head movement corresponding to the input.

  More preferably, the machine learning means includes a time series of predetermined acoustic features extracted from speech at the time of utterance of the utterance subject, an emotion intensity parameter designated with respect to the predetermined emotion of the utterance subject at the time of utterance, From the information indicating the movement of the head of the utterance at the time of utterance, the relationship between the time series and emotion intensity parameters of the predetermined acoustic features and the position or orientation of the head of the utterance in advance is a nonlinear function approximation The function approximation learning means that has already been learned is included.

  According to this head motion synthesis device, head motion synthesis can be performed by nonlinear function approximation when head motion synthesis is performed. Accordingly, it is possible to provide an appropriate head motion synthesis device for synthesizing a nonlinear head motion that cannot be expressed by linear function approximation.

  More preferably, the function approximating learning means includes a time series of predetermined acoustic features extracted from the speech at the time of the utterance of the utterance subject, an emotion intensity parameter designated with respect to the predetermined emotion of the utterance subject at the time of the utterance, From the information indicating the movement of the head of the utterance at the time of the utterance, the relationship between the time series and emotion intensity parameters of the predetermined acoustic features and the position or orientation of the head of the utterance has been learned in advance. Includes neural networks.

  According to this head movement synthesizer, the head movement can be synthesized by nonlinear function approximation using a neural network when the head movement is synthesized. Accordingly, it is possible to provide an appropriate head motion synthesis device for synthesizing a nonlinear head motion that cannot be expressed by linear function approximation without using a neural network.

  The computer program according to the third aspect of the present invention, when executed by a computer, causes the computer to operate as any of the above-described devices. Therefore, it is possible to obtain the same effect as any of the head motion learning device and the head motion synthesis device described above.

  Hereinafter, an embodiment of the invention will be described with reference to the drawings. The present embodiment relates to an apparatus that generates a head movement in a form that reflects not only the acoustic feature quantity of a voice but also an instruction from a user.

<Configuration>
FIG. 2 is a block diagram of an automatic head movement generation system according to an embodiment of the present invention. Referring to FIG. 2, this head movement automatic generation system captures a head movement 60 of a speaker at the time of utterance of learning data, collects head movement data, and a learning 68 A microphone 66 for recording the speech voice 62 of the data speaker, an emotion strength input unit 64 for receiving input of emotion strength specified by the user to clearly indicate the emotion strength, and recording by the microphone 66 Voice storage unit 70 for storing the received voice and emotion intensity parameters input via emotion intensity input unit 64, and head motion for storing data relating to head motion collected by camera 68 A storage unit 72. The voice data stored in the voice storage unit 70 and the data related to the head movement stored in the head movement storage unit 72 are both framed and include common time information. Therefore, it is possible to synchronize both.

  In addition, when a speaker performs an utterance for learning, the speaker performs an emotion in a form corresponding to the emotion intensity input by the emotion intensity input unit 64. From the emotion intensity input unit 64, a parameter indicating the emotion intensity designated for each utterance is input. In this embodiment, the emotion strength uses three stages of “no emotion (normal)”, “includes emotion”, and “includes very strong emotion”. In the present embodiment, these are designated by the values “0”, “0.5”, and “1”, respectively, and stored in the voice storage unit 70.

  The system further calculates a predetermined acoustic feature amount from the voice stored in the voice storage unit 70, and stores the data related to the head movement stored in the head movement storage unit 72 and the voice storage unit 70. Speech-head motion synchronization unit 74 for generating neural network learning data (speech-head motion data) together with emotional intensity specified for speech, and speech-head for the learning A voice-head movement data storage unit 76 for storing movement data, and a neural network using the stored voice-head movement data, a voice, emotional intensity corresponding thereto, and a corresponding head movement A learning unit 78 for learning about the relationship between and a neural network for storing the neural network parameters obtained by the learning And a chromatography click parameter storage unit 80.

  The system further includes a voice storage unit 86 for storing voice recorded in advance based on a script by a voice actor in charge of the character in the animation, and at the time of the utterance of the character based on the script and according to the user's judgment. And an emotion strength setting unit 84 used for the user to input emotion strength parameters. The emotion strength parameters set by the emotion strength setting unit 84 are “0”, “0.5”, and “1”, as in the learning.

  This system further uses the neural network parameters stored in the neural network parameter storage unit 80 to use the acoustic feature amount obtained from the voice stored in the voice storage unit 86 and the emotion set by the emotion strength setting unit 84. A head motion generation unit 82 for generating a head motion 90 of the character based on the intensity parameter.

  Here, the emotion strength is an element for controlling the head movement by the strength of emotion. In this embodiment, it is assumed that “anger emotion strength” is handled. Regarding the emotion of anger, it is possible to consider emotional intensity such as normal state (not angry at all), anger, and rage. As described above, “normal state” is represented by an emotion intensity value (EF value) of 0, “anger” is represented by an EF value of 0.5, and “furious” is represented by an EF value of 1.0. The user can freely express the strength of various emotions of the character according to the situation by arbitrarily selecting the value of the emotion strength (EF). For example, “weak anger” can be expressed by selecting EF value = 0.2. Further, since it is possible to select an EF value of 1.0 or more, for example, an EF value of 1.5 or the like can be selected to express “impossible rage” or the like.

  Moreover, this emotion strength is not limited to anger emotion strength. For example, it is possible to set a joy emotion intensity such as normal state to joy to joy.

  FIG. 3 shows a detailed block diagram of the head movement generation unit 82. Referring to FIG. 3, the head movement generation unit 82 decomposes an input speech signal into phonemes, and outputs a speech feature 110 and a phoneme string corresponding to the input speech. A speech information storage unit 112 for storing the acoustic feature quantity and phoneme sequence output from the speech recognition unit 110, a neural network preset by parameters stored in the neural network parameter storage unit 116, and speech information storage The head motion automatic generation unit 114 for generating a head motion parameter from the acoustic feature amount stored in the unit 112 and the emotion intensity set by the user, and the head generated by the head motion automatic generation unit 114 A head motion parameter storage unit 118 for storing head motion parameters, and a head motion parameter stored in the head motion parameter storage unit 118. Use over data comprises a head behavioral synthesis unit 120 for synthesizing the head motion of the image.

  FIG. 4 shows a block diagram of the head movement automatic generation unit 114 shown in FIG. Referring to FIG. 4, the head movement automatic generation unit 114 calculates the fundamental frequency (F0) 170 corresponding to the voice pitch, the voice frequency from the acoustic feature amount of each frame stored in the voice information storage unit 112. The power 172 corresponding to the size and the utterance time information 174 are extracted and stored in the feature quantity extraction unit 176 and the feature quantity extraction unit 176 for storing the acoustic feature quantity of the immediately preceding 11 frames including the calculated frame. In addition, the sound feature amount for the previous 11 frames and the emotion strength 178 set by the user via the emotion strength setting unit 84 are received as inputs, and the head rotation angles Rx182 and Ry184, which are information related to the head motion of the character. , And a neural network 180 for outputting Rz186. Details of the head rotation angle and utterance time information will be described later. The neural network 180 is preset using the neural network parameters stored in the neural network parameter storage unit 80. Therefore, the neural network 180 can output information on the head movement according to the input acoustic feature quantity and emotion intensity parameter in accordance with the learning by the learning unit 78 shown in FIG.

  FIG. 5 shows details of processing related to learning of the neural network, which is processing performed by the learning unit 78 shown in FIG. Referring to FIG. 5, first, in process 134, learning speech is recorded based on learning text 132. The recorded voice is stored in the storage device 136.

  The process 142 divides the voice stored in the storage device 136 into phonemes. The divided phonemes are sent to the process 152. This phoneme is not used for creating head motion data, but for synthesizing the movement of the mouth of the character.

  In the process 138, the voice stored in the storage device 136 is framed into a frame having a predetermined frame length and a predetermined shift length. The framed audio is sent to processing 140 and processing 144.

  In process 144, an utterance section is detected from the frame obtained in process 138. Various methods can be used to detect the utterance period. Since the recording of the learning text 132 is normally performed in the studio, it is easy to distinguish between the speech section and the silent section. Information specifying the detected utterance period is sent to the process 152.

  In process 140, an acoustic feature quantity including a fundamental frequency and power is calculated from each frame of the framed speech. The calculated acoustic feature amount is sent to the process 152.

  Next, in process 130, an arbitrary emotion intensity is set by the user. The set emotion strength parameter is sent to the process 152.

  On the other hand, in the process 148, the head motion data of the speaker when the learning text 132 is uttered is collected by motion capture using the camera 146. Head movement data obtained by motion capture is stored in the storage device 150. The head movement data stored in the storage device 150 is given to the process 152.

  In process 152, learning data is created after synchronizing the voice and head movement with reference to the emotion intensity, acoustic feature, speech segment, phoneme, and head movement data. That is, learning speech-head motion data is created from the acoustic feature amount of each frame, the designated emotion intensity parameter, and the head motion data corresponding to the frame. This learning data is given to the storage device 154.

  By performing neural network learning based on the voice-head movement data stored in the storage device 154, neural network parameters used when the head movement is automatically generated from the voice can be obtained.

  FIG. 6 shows details of the speech time information. Speaking time information is an acoustic feature to represent head movements that are likely to be related to the elapsed utterance time commonly seen during speech, such as raising the head at the beginning of the talk and lowering the head at the end of the talk. Incorporated as one of the quantities.

  With reference to FIG. 6, the vertical axis represents the utterance time information, and the horizontal axis represents the time from the start to the end of the utterance. An utterance time information of 0 indicates a state when an utterance is started or when no utterance is made. The utterance time information of 1 indicates the end of the utterance.

  FIG. 7 shows details of the head rotation angle. Referring to FIG. 7, as shown in the face image, head rotation angles Rx160, Ry162, and Rz164 are rotations about three axes (x axis, y axis, and z axis) of predetermined three-dimensional coordinates, respectively. Represents an angle. Rx is an angle for use in a vertical movement of the head, that is, a movement that nods or faces upward. Ry is an angle for use in a movement that causes the neck to bend in the left-right direction. Rz is an angle used for the movement of turning the face in the left-right direction. By combining these three axes of rotation angles, head movement can be expressed by three-dimensional rotation.

  In the embodiment of the present invention, the head direction (rotation) according to the head angle is described as an example of the head movement, but it is based on the position of the head, that is, the movement of the head translating. It is possible to synthesize head movements, and it is also possible to synthesize head movements that combine translational motion and rotational motion.

<Operation>
The head movement automatic generation system according to the present embodiment described above operates as follows. There are three phases in the operation of this system. The first phase is a learning phase, the second phase is a voice recording phase based on a script, and the third phase is a phase in which character head movement data is created based on the recorded voice. In addition to this, the entire animation creation system includes a process of designing a character and creating a facial expression of the animation. However, these are not related to the present invention and will not be described here.

  Referring to FIG. 2, at the time of learning, in the head movement automatic generation system, the utterance voice 62 of the speaker at the time of learning is recorded by a microphone 66 synchronized with camera 68 and stored in voice storage unit 70. On the other hand, the head motion data of the speaker recorded by the camera 68 is stored in the head motion storage unit 72. At this time, common time information is attached to both data. Instruct the speaker to speak according to one of the predetermined emotional strengths (in the case of this embodiment, three types of “normal”, “anger”, and “wrath” regarding anger) Is issued. Then, the emotion strength input unit 64 manually inputs an emotion strength parameter (any one of “0”, “0.5”, and “1”) indicating the emotion strength. This emotion strength parameter is stored in the voice storage unit 70 together with the corresponding utterance voice.

  The voice and emotion strength stored in the voice storage unit 70 and the data related to the head motion stored in the head motion storage unit 72 are both framed, and voice data and emotion strength parameters corresponding to frames at the same time The head movement parameters are combined into one data by the voice-head movement synchronization unit 74. In this way, speech-head motion data for learning is created and stored in the speech-head motion data storage unit 76.

  Subsequently, using the speech-head motion data stored in the speech-head motion data storage unit 76, the learning unit 78 learns the relationship between the speech and emotion intensity and the corresponding head motion in a neural network. Let The neural network parameters obtained by the learning are stored in the neural network parameter storage unit 80.

  At the time of recording the script, the voice actor utters the speech of a predetermined character while watching the script. This voice is recorded and stored in the voice storage unit 86 shown in FIG.

  At the time of head motion creation, the head motion generation unit 82 automatically generates head motion from speech using the neural network parameters obtained at the time of learning. Specifically, first, a neural network is set using parameters stored in the neural network parameter storage unit 80. Further, the user sets any emotion strength parameter of 0, 0.5, or 1 using the emotion strength setting unit 84 for each utterance. The head motion generation unit 82 extracts the acoustic feature amount (F0, power, and utterance time information) from the voice read from the voice storage unit 86, and the acoustic feature amount and the emotion set for the utterance by the user Using the intensity parameter, create and give input to the neural network. In response to this input, the head movement 90 of the character is automatically generated by the neural network.

  Details of the operation of the head movement generation unit 82 will be described with reference to FIG. First, the speech data stored in the speech storage unit 86 is given to the speech recognition unit 110, and is output as a phoneme string that is decomposed into phonemes by the speech recognition unit 110 and to which acoustic features are added. The acoustic feature amount and the phoneme string are stored in the voice information storage unit 112.

  Among the audio information stored in the audio information storage unit 112, the one related to the acoustic feature amount is output to the head movement automatic generation unit 114 in order to generate the head movement. The phoneme string is used to synthesize the mouth movement of the character.

  On the other hand, the emotion strength parameter set by the user using the emotion strength setting unit 84 is also given to the head motion automatic generation unit 114.

  In the head movement automatic generation unit 114, input data to the neural network is created from the acoustic feature amount given from the voice information storage unit 112 and the emotion strength set by the emotion strength setting unit 84. For this input data, the last 11 frames of the audio data are stored. The input data for these 11 frames is given as an input to the neural network. In response, the corresponding head movement parameters are output from the neural network. Specific conversion processing in the head movement automatic generation unit 114 will be described later.

  The head movement parameter output from the head movement automatic generation unit 114 is stored in the head movement parameter storage unit 118 shown in FIG. The head motion synthesis unit 120 synthesizes the head motion of the image using the head motion parameters stored in the head motion parameter storage unit 118.

  The head movement automatic generation unit 114 operates as follows.

  Referring to FIG. 4, first, of the information regarding the acoustic feature amount of each frame output from the speech information storage unit 112, the voice height 170, the voice volume 172, and the utterance time information 174 are feature amounts. Extracted by the extraction unit 176. This information is held in the feature amount extraction unit 176 for the previous 11 frames. The feature amount for the immediately previous 11 frames and the emotion strength arbitrarily set by the user in the emotion strength setting unit 84 are given to the neural network 180. In response to this, the neural network 180 outputs head motion parameters (head rotation angles Rx182, Ry184, and Rz186), which are parameters for synthesizing the head motion. The neural network 180 is preset by the neural network parameters stored in the neural network parameter storage unit 80. This parameter indicates the relationship between input (speech) and output (head motion parameter) in the neural network 180 according to the learning result. The output head motion parameter is stored in the head motion parameter storage unit 118.

<Specific example of head motion generation>
FIG. 8 shows specific examples of head movements that change by inputting various emotional intensities.

  Referring to FIG. 8, the vertical axis represents one of the head rotation angles, the value of angle Rx representing the vertical movement of the head, and the horizontal axis represents the time from the start of speech (speech time information). In units of 1/1000 seconds. The English sentence written in the graph is the utterance sentence used in this specific example. In this example, anger emotion strength was used as emotion strength. In addition, the anger emotion intensity was changed from 0.0 (normal state) to 1.0 (furious) in steps of 0.2.

  As shown in FIG. 8, when the emotion intensity is changed, the vertical movement of the head indicated by the waveform changes. Then, as the emotion intensity increases, that is, as the “rage” approaches, the vertical movement of the head increases from 0.75 seconds to 1.3 seconds. That is, the movement of the head image becomes intense.

  In FIG. 8, the value of the corner Rx, particularly in the vicinity of 1 second, changes non-linearly with respect to the emotion intensity value. This is characteristic when nonlinear conversion is performed like a neural network, and the head movement of the speaker actually shows such nonlinearity. Unlike neural networks, when head movements are generated using a linear method, the nonlinearity of this vertical movement is lost, and the magnitude of movement changes only to linear with respect to emotional intensity. Only a monotonous deformation result can be obtained.

<Experiment for performance evaluation>
In order to evaluate the effectiveness of the device for generating head motion according to the present embodiment, the head rotation angle Rx of the actual head, which is a part of the data used for learning, and the head motion are FIG. 9 shows the result of comparison with the head rotation angle Rx when generated by the system according to the present embodiment. In FIG. 9, the vertical axis represents the head rotation angle Rx, and the horizontal axis represents the time from the start of utterance in units of 1/1000 seconds. The emotional strength used in this performance evaluation experiment is anger emotional strength.

  As shown in FIG. 9, normal state (upper part of FIG. 9: emotion intensity EF = 0.0), anger (middle part of FIG. 9: EF = 0.5), furious (lower part of FIG. 9: EF = 1) .0), the waveforms of the rotation angle Rx of the actual head movement and the rotation angle Rx of the head movement generated by the system according to the present embodiment are very similar to each other.

  The head rotation angle generated using the acoustic feature amount obtained from the learning speech prepared in advance and the emotion intensity set by the user from the similarities of the waveforms described above is in any emotion intensity. It can also be seen that it is similar to the head rotation angle of the actual head movement used for learning. From this result, it can be said that even if the user arbitrarily sets the emotion strength, it is expected that a natural head movement synchronized with the voice is generated. Therefore, when a head motion is generated from a voice using a neural network, the user inputs an arbitrary emotion intensity, so that a natural head motion can be generated according to the user's sensitivity. .

  As described above, when the head motion is automatically generated from the voice, the natural head motion can be generated from the voice according to the emotion strength by using the emotion strength arbitrarily set by the user. By generating the head movement by such a method, it is possible to generate the head movement efficiently according to the sensitivity of the user.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim in the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

It is a figure which shows the head movement automatic generation system by a prior art. It is a figure which shows the head movement automatic generation system which concerns on this invention. It is a figure which shows the detail of a head movement production | generation part. It is a figure which shows the detail of a head movement automatic generation part. It is a figure shown about the detail of the process which concerns on the learning method of a neural network. It is a figure shown about the detail of speech time information. It is a figure which shows the detail of a head rotation angle. It is a figure which shows the specific example of the head movement which changes by inputting various emotion intensity | strengths. It is a figure which shows the result of the performance evaluation of this invention.

Explanation of symbols

64 emotion intensity input unit 74 voice-head motion synchronization unit 78 learning unit 84 emotion strength setting unit 86 voice storage unit 114 head motion automatic generation unit 148 motion capture 176 feature quantity extraction unit 180 neural network

Claims (11)

A head movement learning device for learning the relationship between an utterance and the movement of the head of the utterance subject accompanying the utterance by machine learning,
An emotion strength input means for receiving an input of an emotion strength parameter indicating the strength related to a predetermined emotion of the utterance subject at the time of utterance;
A predetermined acoustic feature amount extracted as a time series from the speech of the utterance at the time of each utterance, the emotion strength parameter input via the emotion strength input means regarding the utterance, and And learning means for learning a relationship between the predetermined acoustic feature amount and the emotion intensity parameter and the movement of the head of the utterance subject from information indicating the movement of the head of the utterance subject. , Head movement learning device. The learning means includes
An acoustic feature quantity extracting means for receiving the voice at the time of the utterance of the utterance subject and extracting the acoustic feature quantity of the voice every predetermined time from the start of the utterance;
Time information giving means for attaching information indicating the time from the start of the utterance to the acoustic feature quantity extracted by the acoustic feature quantity extraction means;
Head position information acquisition means for acquiring information indicating the position or orientation of the head of the utterance in association with the time of the utterance subject,
For an utterance, the acoustic feature amount extracted by the acoustic feature amount extraction unit and the time information from the utterance added by the time information addition unit, and the emotion strength parameter input by the emotion strength input unit for the utterance A synchronization means for generating learning data by synchronizing the position or orientation of the head of the utterance subject acquired by the head position information acquisition means;
For learning the relationship between the time series of the acoustic feature quantity and the emotion intensity parameter and the change in the position or orientation of the head of the utterance subject using the learning data generated by the synchronization means The head movement learning apparatus according to claim 1, comprising: means.   The means for learning uses the learning data generated by the synchronization means, the time series of the predetermined acoustic feature amount given the time from the start of the utterance of the certain utterance, the emotion intensity parameter, and the The head movement learning device according to claim 2, further comprising a function approximation learning means for learning a relationship between the position or orientation of the head of the utterance subject by a predetermined nonlinear function approximation.   The function approximation learning means learns the relationship between the acoustic feature value and the emotion intensity parameter and the position or orientation of the head of the utterance subject using the learning data generated by the synchronization means as learning data. The head movement learning device according to claim 3, comprising a neural network for performing the operation. A head motion synthesizer for synthesizing the movement of the head of the utterance subject image accompanying the utterance from the utterance,
Given the time series of predetermined acoustic features extracted from the speech of the utterance at the time of utterance and the emotion intensity parameter specified for the utterance, the head movement of the utterance at the time of utterance of the speech is estimated Head position estimating means for
Acoustic feature quantity extraction means for extracting a time series of the predetermined acoustic feature quantity from speech;
The utterance subject synchronized with the voice by providing the head position estimation unit with the time series of the predetermined acoustic feature amount extracted by the acoustic feature amount extraction unit and the designated emotion strength parameter And a head motion generating means for obtaining information relating to the head motion of the head as a series of outputs from the head position estimating means.   The head motion synthesizer according to claim 5, further comprising means for storing voice in advance and supplying the sound to the acoustic feature quantity extraction means.   The head movement synthesizer according to claim 5 or 6, further comprising means for giving an emotion intensity parameter corresponding to an emotion intensity input by a user to the head movement generation means.   The head position estimation means includes a time series of predetermined acoustic features extracted from speech at the time of the utterance of the utterance subject, and an emotion intensity parameter designated for the predetermined emotion of the utterance subject at the time of utterance. Between the time series of the predetermined acoustic feature amount and the emotion intensity parameter and the position or orientation of the head of the utterance subject from the information indicating the movement of the head of the utterance subject during the utterance The head motion synthesizer according to any one of claims 5 to 7, further comprising machine learning means that has been previously learned.   The machine learning means includes a time series of predetermined acoustic features extracted from speech at the time of the utterance of the utterance subject, an emotion intensity parameter designated with respect to the predetermined emotion of the utterance subject at the time of the utterance, and the utterance From the information indicating the movement of the head of the utterance subject at the time, the relationship between the time series of the predetermined acoustic feature amount and the emotion intensity parameter and the position or orientation of the head of the utterance subject is nonlinear in advance The head movement synthesizer according to claim 8, comprising function approximation learning means learned by function approximation.   The function approximation learning means includes a time series of predetermined acoustic features extracted from speech at the time of utterance of the utterance subject, an emotion intensity parameter designated with respect to the predetermined emotion of the utterance subject at the time of utterance, Learning in advance the relationship between the time series of the predetermined acoustic feature quantity and the emotion intensity parameter and the position or orientation of the head of the utterance subject from information indicating the movement of the head of the utterance subject during utterance The head movement synthesis device according to claim 9, comprising a completed neural network.   A computer program that, when executed by a computer, causes the computer to operate as the apparatus according to any one of claims 1 to 11.

Images