JP5164911B2 - Avatar generating apparatus, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an avatar having natural expression and facilitates speech. <P>SOLUTION: The avatar generating device is provided with: a means 104 for predicting a break of expiration paragraphs of speech voice; a means 105 for calculating a laughter estimation value by a first model which outputs a larger laughter estimation value as a basic frequency is large according to the basic frequency of the speech voice in an expiration paragraph immediately before the break; a means 105 for setting a laughter level according to whether or not the laughter estimation value is a first threshold or more; a means 106 for calculating a smile estimation value which is the sum or the product of a first output value of a second model which uses the last smile estimation value as input, a laughter estimation value immediately before, and a second output value of a third model which uses sound pressure in a predetermined period at the beginning of the next expiration paragraph as input; a means 105 for setting a smile level according to whether or not the smile estimation value is a second threshold or more; and means 108, 109 for generating an avatar according to the laughter level and the smile level. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  In the present invention, when a voice mail, video mail, videophone, etc. are spoken to a device or recorded, an avatar prompts the speaker to speak in accordance with the voice of the speaker, The present invention relates to a voice recording interface for enabling a speaker to blow a voice or to send an emotional message by returning laughter, and a technique related to a smile / laughter guess model.

  Traditional real-time and non-real-time media communication (communication with a machine between people) has problems such as being difficult to grasp, anxious, and nervous because it speaks to a machine that does not respond .

  In order to solve this, there is a research example (for example, refer to Patent Document 1 and Patent Document 2) that attempts to lower the above-described barrier by using an avatar when communicating, but the user does not operate the avatar with keys. There is a trouble of not being. In non-real-time communication, there is a problem that an avatar cannot be operated interactively for one utterance content.

JP 2001-160154 A JP 2005-327096 A

  As described above, in the conventional communication support system, the user has to operate a reaction such as an avatar's facial expression or gesture. Therefore, it is difficult to return a natural and interactive response, and it is not used for non-real-time communication.

  Accordingly, an object of the present invention is to provide an avatar generation apparatus, method, and program for generating such an avatar that solves such a conventional problem and has a natural expression and changes so as to promote speech.

In order to solve the above-described problems, an avatar generating apparatus according to the present invention includes a prediction unit that predicts a break in an expiratory paragraph from a time-series relationship of the utterance voice, and a basic of the utterance voice in the expiratory paragraph immediately before the break. First estimation means for calculating a laughter estimated value corresponding to the current break using a first statistical model that outputs a larger laughter estimated value as the fundamental frequency increases, and the laughter estimated value is greater than or equal to a first threshold value A first setting means for setting a laughing level based on whether or not, a first output value of a second statistical model with a smile estimated value calculated at the previous break as an input, and a laughter estimated value calculated at the current break , And a smile estimated value, which is the sum or product of the second output values of the third statistical model with the sound pressure in the predetermined period at the beginning of the exhalation paragraph following the current break as an input, is calculated according to the current break With second estimation means Comprising: a second setting means for setting a smile level based on whether the estimated smile value is equal to or greater than a second threshold; and a generating means for generating an avatar in accordance with the laughter level and the smile level. To do.

  According to the avatar generation device, method, and program of the present invention, it is possible to display so as to generate an avatar whose expression is natural and changes so as to promote speech.

The block diagram of the avatar production | generation apparatus of embodiment of this invention. The flowchart which shows an example of operation | movement of the avatar production | generation apparatus of FIG. The figure which shows the example of the facial expression of the avatar produced | generated by the avatar production | generation part of FIG.

Hereinafter, an avatar generating device, method, and program according to embodiments of the present invention will be described in detail with reference to the drawings. Note that, in the following embodiments, the same numbered portions are assumed to perform the same operation, and repeated description is omitted.
First, an outline of an embodiment of the present invention will be described.
Human laughter is broadly divided into voluntary laughter and involuntary laughter. In this embodiment, social laughter is modeled in optional laughter. Social laughter is a smile used to keep a good relationship with the other person when communicating, to make it easier for the counselor to talk, or when trying to build a good relationship with the person you meet for the first time Seen.

  In this embodiment, among the social laughter performed by the listener of the story, the laughter (smile of interest) that expresses what he / she listens with interest and encourages utterance, and the positive that the utterance is interesting or in favor. Model the laughter (evaluation laughter) that expresses the evaluation. The smile of interest occurs before or shortly after the speaker speaks, and is moderately moderate in duration. On the other hand, evaluation laughter occurs at or just before the end of a mass of utterances (exhalation paragraph), which is stronger than the smile of interest, and continues until the next utterance begins or begins. In the present embodiment, the facial expression of the listener is estimated according to the voice of the speaker.

  In the embodiment of the present invention, when speaking or recording toward a device, the avatar (corresponding to the listener) returns a smile or laughter in response to the uttered voice, making it easier to record the voice or expressiveness. In order to make it easier for the speaker to speak, the avatar returns a smile or laughter response when the estimated value of smile or laughter exceeds a predetermined threshold. Note that an avatar is one expression of a user as a person or an alternation, indicates an object that represents the user's incarnation, and is created by computer graphics.

Next, an avatar generating apparatus according to an embodiment of the present invention will be described with reference to FIG.
The avatar generation device of this embodiment includes a microphone 101, a personal computer PC102, and a display 110. The PC102 includes a reaction estimation unit 103, an avatar expression determination unit 108, and an avatar generation unit 109. The response estimation unit 103 includes an exhalation paragraph prediction unit 104, a laughter estimation unit 105, and a smile estimation unit 106, and the smile estimation unit 106 includes a memory 107.
A device that does not include the microphone 101 and the display 110 and includes the reaction estimation unit 103, the avatar expression determination unit 108, and the avatar generation unit 109 may be referred to as an avatar generation device.

The microphone 101 acquires the voice (voice packet) of the speaker.
The expiratory paragraph prediction unit 104 predicts a break in the expiratory paragraph from the time-series relationship between on and off of the voice acquired by the microphone 101. Specifically, the expiratory paragraph prediction unit 104 predicts the end timing of the expiratory paragraph in real time from the on / off rhythm of the speech using a statistical model. Here, the exhalation paragraph indicates a period from one breath change to the next breath change. The statistical model is, for example, an MA (moving average) model or an HMM (hidden Markov model). In addition, although a statistical model is used with the same meaning hereafter, it is used by various calculations and each may be a different model. Also, if the calculation contents are different, different parameters are set even in the same model.

The laughter estimation unit 105 models the “evaluation laughter” described above and determines a laughter level. Specifically, the laughter estimation unit 105 estimates the laughter by a statistical model using the fundamental frequency of the uttered speech in the immediately preceding exhalation paragraph when there is a break in the exhalation paragraph at the timing predicted by the exhalation paragraph prediction unit 104 A value is determined and this estimated laughter value is compared with a laughter threshold. The fundamental frequency indicates the highest frequency among the frequencies that exist after converting the sound pressure of the uttered voice acquired by the microphone 101 into a frequency and removing noise. For example, the laughter estimation unit 105 calculates the fundamental frequency from the uttered voice acquired by the microphone 101.
In the statistical model, the larger the fundamental frequency, the larger the estimated laughter value that is the output value of the statistical model. The laughter estimation unit 105 stores L laughter threshold values (L is an integer equal to or greater than 1), determines whether the estimated laughter value is equal to or greater than these threshold values, and sets a laughter level corresponding to the estimated laughter value. . The laughter estimation unit 105 sets, for example, L = 2, “laughter threshold 2> laughter threshold 1”, sets the laughter level to the laughter level 2 when the estimated laughter is equal to or higher than the laughter threshold 2, and the estimated laughter If it is smaller than the threshold value 2 and greater than or equal to the laughter threshold value 1, the laughter level is set to the laughter level 1, and these set values are passed to the avatar expression determination unit 108. On the other hand, when the estimated laughter value is smaller than the laughter threshold value 1, the laughter level is not set and the fact is passed to the avatar facial expression determination unit 108.

The smile estimation unit 106 models the above-mentioned “smile of interest” to determine a smile level. Specifically, the smile estimation unit 106 outputs the output value of the statistical model having the immediately preceding M smile estimate values (M is an integer of 1 or more) as input, the immediately preceding laugh estimate value, and the beginning of the next exhalation paragraph. The estimated smile value is obtained from the output value of the statistical model with the sound pressure of a predetermined period (for example, 500 ms) as an input. For example, the smile estimation unit 106 outputs an output value of a statistical model having M previous estimated smile values (M is an integer of 1 or more) as input, a previous laughter estimated value, and a predetermined value at the beginning of the next exhalation paragraph. The sum or product of the output value of the statistical model with the sound pressure of the period as an input is used as the estimated smile value. The statistical model outputs a larger output value as the previous M estimated smile values are larger, and outputs a larger output value as the sound pressure for a predetermined period is larger. The previous M estimated smile values indicate smile estimates in each of the exhalation paragraphs up to M before the current exhalation paragraph. The immediately preceding estimated laughter value indicates the estimated laughter value in the exhalation paragraph immediately before the current exhalation paragraph. The memory 107 stores the immediately preceding M smiling estimated values and the immediately preceding laughing estimated value.
The smile estimation unit 106 stores N smile threshold values (N is an integer equal to or greater than 1), determines whether the smile estimate value is greater than or equal to these threshold values, and sets a smile level corresponding to the smile estimate value. . The smile estimation unit 106 sets, for example, N = 3, “smile threshold 3> smile threshold 2> smile threshold 1”, sets the smile level to smile level 3 when the smile estimate is equal to or greater than smile threshold 3, and smiles. If the estimated value is smaller than the smile threshold 3 and greater than or equal to the smile threshold 2, the smile level is set to the smile level 2. If the estimated smile is smaller than the smile threshold 2 and greater than or equal to the smile threshold 1, the smile is set. The level is set to smile level 1, and these set values are passed to avatar facial expression determination section 108. On the other hand, when the estimated smile value is smaller than the smile threshold 1, it is determined that there is no laughter and no smile, and that effect is passed to the avatar facial expression determination unit 108.

The avatar facial expression determination unit 108 receives the laughing level and the smiling level from the laughing estimation unit 105 and the smile estimation unit 106, respectively. Select facial expressions according to each level. As described above, for example, when L = 2 and N = 3, one of the six facial expressions is selected (see FIG. 3).
The avatar generation unit 109 generates an avatar (laughing avatar) indicating a listener's smile or laughter according to the voice of the speaker. At this time, the expression of the avatar is based on the information received from the avatar expression determination unit 108. The display 110 displays the avatar image 151 generated by the avatar generation unit 109.

Next, an example of the operation of the avatar generation device in FIG. 1 will be described with reference to FIG.
First, the microphone 101 acquires a voice packet (step S202). Next, the expiratory paragraph prediction unit 104 predicts a break in the expiratory paragraph by the expiratory paragraph predicting process based on the statistical model (step S203). The laughing estimation unit 105 proceeds to step S205 if it is a break in the exhalation paragraph, and returns to step S202 if it is not a break in the exhalation paragraph. The laughter estimation unit 105 obtains an estimated value of laughter by a statistical model using the fundamental frequency of the uttered voice in the previous exhalation paragraph (step S205). The laughter estimation unit 105 determines whether or not the estimated laughter value is greater than or equal to the laughter threshold value 2 (step S206). If the estimated laughter value is equal to or greater than the laughter threshold value 2, the laughter level is set to 2 (step S212). When it falls below, it progresses to step S207. The laughter estimation unit 105 determines whether or not the estimated laughter value is greater than or equal to the laughter threshold value 1 (step S207). If the estimated laughter value is greater than or equal to the laughter threshold value 1, the laughter level is set to 1 (step S213). If it falls, the process proceeds to processing 208. At this time, “laughter threshold 2> laughter threshold 1”. When the laughter level is determined in step S212 and step S213, the avatar facial expression determination unit 108, the avatar generation unit 109, and the display 110 present laughter according to the level (step S217).

A statistical model in which the smile estimation unit 106 inputs the three estimated smile values immediately before and outputs the degree to which the current facial expression is likely to laugh (the larger the input value, the larger the value that is output and the degree of smile The estimated smile value is calculated as the sum or product of the output value of the statistic model with the input value of the previous laughter estimate value and the 500 ms sound pressure at the beginning of the next exhalation paragraph (statistic model). Then, it is determined whether or not the listener is smiling by voice) (step S208). The smile estimation unit 106 determines whether or not the smile estimation value is equal to or greater than the smile threshold 3 (step S209). If the smile estimation value is the smile threshold 3, the smile level is set to 3 (step S214) and falls below the smile threshold 3. If YES, go to step S210. The smile estimation unit 106 determines whether or not the smile estimation value is equal to or greater than the smile threshold 2 (step S210). If the smile estimation value is equal to or greater than the smile threshold 2, the smile level is set to 2 (step S215), and the smile threshold 2 is set. When it falls below, it progresses to step S211. The smile estimation unit 106 determines whether the smile estimation value is equal to or greater than the smile threshold 1 (step S211). If the smile estimation value is equal to or greater than the smile threshold 1, the smile level is set to 1 (step S216), and the smile threshold 1 is set. When it falls below, it progresses to step S219. The avatar facial expression determination unit 108, the avatar generation unit 109, and the display 110 present a facial expression without laughter and smile (step S219). When the smile level is determined in steps S214, S215, and S216, a smile according to the level is presented. (Step S218). (However, smile threshold 3> smile threshold 2> smile threshold 1)
The avatar facial expression determination unit 108 determines the avatar facial expression to be presented according to each level as shown in Table 1, for example. Also, the avatar generation unit 109 generates, for example, the avatar shown in FIG.

  According to the embodiment described above, a microphone for inputting a speaker's uttered voice is prepared, and the voice captured from the microphone is sent to the laughing estimation unit and the smile estimation unit in the PC, and the estimated laughter value and the smile The estimated value can be calculated based on the statistical model, and laughter and smile can be determined based on the obtained estimated value, and the avatar displayed on the display can express laughter or smile. As a result, the avatar that assumes the listener according to the utterance rhythm of the speaker naturally returns a smile or laughter, so that a pulling phenomenon occurs between the speaker and the avatar. Specifically, the avatar feels as if listening while affirming the story, making it easier for the speaker to speak, and returning the laughter that pulls out according to the chronological rhythm of the utterance, There is an other party, and it feels that he / she listens, so it has the effect of feeling emotional in the speaker's message.

  In addition, a prediction model is created from the chronological on / off rhythm of the speaker's utterance to determine the occurrence timing of smiles and laughter, and a guess model is established from the fundamental frequency and sound pressure of the spoken speech to laugh and smile. When it occurs, it is possible to provide a voice recording interface that makes it easy for a speaker to speak and a message with emotions, and a model for estimating a smile / laugh.

  In addition, avatars return a natural and exciting smile and laughter to the speaker's utterances, helping them speak and record to devices such as voicemail, videomail, and videophone.

  In addition, when speaking toward the device or recording, the avatar returns a smile or laughter in response to the spoken voice, making it easier to record the voice or to express an expressive message. it can. Specifically, the avatar displayed on the display returns the laughter and smile used for pulling in the communication of people in real time to the utterance voice of the speaker, so that the speaker can listen favorably. It becomes easy to speak with a sense of security. Also, in face-to-face communication between people, the smiles that they make affect each other and make each other smile (synchronization), and the communication rises, but when the avatar returns a smile to the speech, It makes you smile. The speaker feels that the avatar is also synchronized, and the utterance rises, and emotions can be put into the speech.

  Furthermore, the avatar generation device of the present embodiment can be incorporated into various systems that blow sound. As a specific example, the avatar generating device of this embodiment is generally used when voice is blown into voice mail, answering machine, video message, etc. By presenting an avatar that returns a smile / laugh according to the utterance voice, the speaker can calm down, inject a message with a good tempo and emotion.

  It can also be applied to real-time voice communication media. Because the other party's face cannot be seen on the phone, there are no clues such as facial expressions that can be observed in face-to-face communication, so the other party is listening properly, understands the conversation, or agrees with the conversation I have anxiety that I don't know. By displaying the avatar of the present invention on the display of each other's terminal when making a call with an IP phone or a mobile phone, each user can talk to the avatar as if it were a call partner. , I can speak with a sense of security that the other party is listening, understanding the content, and agreeing with the story. As a result, you can speak emotionally and the conversation will rise.

  Furthermore, application to a TV conference is also possible. In the video conference, the state of the other party in the remote place that has a conversation is displayed on the display, so you can observe the expression and nodding. However, in video conferences, there is little sense of sharing the same space as the other party, the other party can see, but the line of sight does not match, so eye contact cannot be taken, and it is difficult to understand whether the message is transmitted. As a result, in the video conference, it is impossible to perform mutual pull-in in communication originally performed by a person. Therefore, the system of the present invention is displayed beside the display in front of each participant. By reacting the avatar to the speech of all participants and sharing it with all the participants, the participants can speak with confidence and emotionally, and the avatar acts as a bridge to engage each other. It can be carried out.

  Furthermore, the avatar generating apparatus of the present embodiment is also effective for reading voices, announcements, and automatic voice responses that are not limited to messages. Since these voice injections are performed in a situation where there is no other party, the tempo of the speech depends on the rhythm of the speaker alone. In a face-to-face environment, a message is issued on a mutual rhythm that includes not only the sender but also the receiver, but if it depends on the rhythm of the speaker alone, the message becomes mechanical. Therefore, by displaying the avatar with the avatar generating apparatus of the present invention and blowing a message on the avatar, it is possible to record a message with human feeling and expression.

The instructions shown in the processing procedure shown in the above embodiment can be executed based on a program that is software. The general-purpose computer system stores this program in advance and reads this program, so that the same effect as that obtained by the avatar generating apparatus of the above-described embodiment can be obtained. The instructions described in the above-described embodiments are, as programs that can be executed by a computer, magnetic disks (flexible disks, hard disks, etc.), optical disks (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD). ± R, DVD ± RW, etc.), semiconductor memory, or a similar recording medium. As long as the recording medium is readable by the computer or the embedded system, the storage format may be any form. If the computer reads the program from the recording medium and causes the CPU to execute instructions described in the program based on the program, the same operation as the avatar generation device of the above-described embodiment can be realized. Of course, when the computer acquires or reads the program, it may be acquired or read through a network.
In addition, the OS (operating system), database management software, MW (middleware) such as a network, etc. running on the computer based on the instructions of the program installed in the computer or embedded system from the recording medium implement this embodiment. A part of each process for performing may be executed.
Furthermore, the recording medium in the present invention is not limited to a medium independent of a computer or an embedded system, but also includes a recording medium in which a program transmitted via a LAN or the Internet is downloaded and stored or temporarily stored.
Further, the number of recording media is not limited to one, and when the processing in the present embodiment is executed from a plurality of media, it is included in the recording media in the present invention, and the configuration of the media may be any configuration.

The computer or the embedded system in the present invention is for executing each process in the present embodiment based on a program stored in a recording medium, and includes a single device such as a personal computer or a microcomputer, Any configuration such as a system in which apparatuses are connected to a network may be used.
Further, the computer in the embodiment of the present invention is not limited to a personal computer, but includes an arithmetic processing device, a microcomputer, and the like included in an information processing device, and a device capable of realizing the functions in the embodiment of the present invention by a program, The device is a general term.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 101 ... Microphone, 102 ... PC, 103 ... Reaction estimation part, 104 ... Expiration paragraph prediction part, 105 ... Laughter estimation part, 106 ... Smile estimation part, 107 ... Memory 108 avatar expression determining unit 109 avatar generating unit 110 display 151 avatar image.

Claims (5)

A prediction means for predicting the break of the exhalation paragraph from the time-series relationship of the utterance voice;
A first statistic model that outputs a larger estimated laughter value as the fundamental frequency increases in accordance with the fundamental frequency of the speech voice in the exhalation paragraph immediately before the break, and calculates the estimated laughter value corresponding to the current break . An estimation means;
First setting means for setting a laughing level based on whether the estimated laughter value is equal to or greater than a first threshold;
The first output value of the second statistical model with the estimated smile value calculated at the previous break as an input, the estimated laughter value calculated at the current break , and the predetermined period at the beginning of the exhalation paragraph following the current break Second estimation means for calculating a smile estimated value, which is the sum or product of the second output values of the third statistical model with the sound pressure of
Second setting means for setting a smile level based on whether the estimated smile value is equal to or greater than a second threshold;
An avatar generating device comprising: generating means for generating an avatar according to the laughter level and the smile level. The predicting means predicts the break using the first MA model from the on / off rhythm of the speech,
The first estimating means calculates the estimated laughter value by a second MA model,
The first setting means sets the laughter level to laughter level 1 when the estimated laughter value is equal to or greater than the first threshold value,
The generating means generates an avatar corresponding to the laughter level 1,
The second estimating means calculates the smile estimated value by a third MA model,
The second setting means sets the smile level to a smile level 1 when the estimated smile value is greater than or equal to the second threshold value,
The generating means generates an avatar corresponding to the smile level 1;
The second setting means sets the smile level to a level at which no smile and no smile when the estimated smile value is less than the second threshold;
The avatar generation device according to claim 1, wherein the generation unit generates an avatar without laughter and smile corresponding to the level without laughter and smile. A prediction step for predicting a break in an exhalation paragraph from a time-series relationship of speech speech;
A first statistic model that outputs a larger estimated laughter value as the fundamental frequency increases in accordance with the fundamental frequency of the speech voice in the exhalation paragraph immediately before the break, and calculates the estimated laughter value corresponding to the current break . An estimation step;
A first setting step of setting a laughter level based on whether the estimated laughter value is greater than or equal to a first threshold;
The first output value of the second statistical model with the estimated smile value calculated at the previous break as an input, the estimated laughter value calculated at the current break , and the predetermined period at the beginning of the exhalation paragraph following the current break the third and second estimation step of calculating in accordance with the estimated value smile is the sum or the product of the second output value of the statistical model to the current cut that receives the sound pressure,
A second setting step for setting a smile level based on whether the estimated smile value is equal to or greater than a second threshold;
A generation step of generating an avatar according to the laughter level and the smile level. In the predicting step, the cut is predicted using the first MA model from the on / off rhythm of the speech,
In the first estimation step, the estimated laughter value is calculated by a second MA model,
In the first setting step, when the estimated laughter value is greater than or equal to the first threshold, the laughter level is set to laughter level 1,
In the generation step, an avatar corresponding to the laughter level 1 is generated,
In the second estimating step, the estimated smile value is calculated by a third MA model,
In the second setting step, when the estimated smile value is equal to or greater than the second threshold, the smile level is set to a smile level 1,
In the generation step, an avatar corresponding to the smile level 1 is generated,
In the second setting step, when the estimated smile value is less than the second threshold value, the smile level is set to a level that does not laugh and smile;
4. The avatar generation method according to claim 3, wherein in the generation step, an avatar without laughter and smile corresponding to the level without laughter and smile is generated.   The program for making a computer perform each process of the avatar production | generation method of Claim 3 or Claim 4.

Images