JP2005091463A - Information processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information processing device in which speaker's feeling is displayed together with his or her expression. <P>SOLUTION: This information processing device is provided with a display part (240) for displaying the speaker's expression, analysis parts (110, 220) for analyzing the feeling from the speech information of the speaker, and a control part (110) for instructing the display part to display the result of the analysis. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an information processing device, and more particularly, to an information processing device that displays a speaker's facial expression along with emotions.

  Video display devices that display live-action images of utterance scenes are used in systems such as TV telephones and TV conferences. Here, when displaying images of a plurality of persons, the faces of all the persons do not always appear in the image with a sufficient size. For example, in a conference with many people, some attendees may turn their backs to the camera. In this way, the face may not appear in the video or may appear small even if it appears. That is, there are cases where the person who sees the video cannot visually recognize what person in the video is talking about.

For this reason, in the conventional video display device, first, a main video showing the status of the TV conference is displayed. In addition to this, an animation image expressing a person's utterance situation is generated by deforming a two-dimensional face model and displayed as auxiliary information (who is talking) of the main image. (For example, refer to Patent Document 1).
JP 2002-150317 A (pages 2 to 4, FIG. 2)

However, even if you know who is currently speaking, it is difficult to read the speaker's emotions from the TV screen. This is even more so if the screen on which the speaker's facial expression is projected is small.

  The objective of this invention is providing the information processing apparatus which displays the emotion with a speaker.

  1st invention is provided with the display part which displays a speaker's facial expression, the analysis part which analyzes an emotion from a speaker's audio | voice information, and the control part which instruct | indicates that the said analysis result is also displayed on the said display part Information processing apparatus.

  A second invention is the information processing apparatus according to the first invention, wherein the analysis unit analyzes an emotion from a change in frequency of the voice information.

  A third invention is the information processing apparatus according to the first invention, wherein the analysis unit analyzes an emotion from a change in volume of the voice information.

  A fourth invention is the information processing apparatus according to the first invention, wherein the analysis unit analyzes an emotion from a change in a conversation interval of the voice information.

  A fifth invention is the information processing apparatus according to the first invention, wherein the analysis result is a mark indicating the emotion of the speaker.

  ADVANTAGE OF THE INVENTION According to this invention, the information processing apparatus which displays the emotion with the expression of a speaker can be provided.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a schematic block diagram of an information processing apparatus according to an embodiment of the present invention. The MPU 110 starts application software for the TV phone system from the ROM 210. Then, information from the other party (speaker) of the TV phone is acquired from the network connection unit 120. In addition to the facial expression and voice information of the other party, this information includes IP address information for specifying who is currently speaking.

  The other party's facial expression is displayed on the display 240 via the RAM 130 and the image output unit 230 as the main video. The other party's voice is decoded and output from the speaker 260 via the RAM 130 and the sound output unit 250.

  On the other hand, the user's facial expression passes through the camera 140 and the image input unit 150, the voice passes through the microphone 160 and the voice input unit 170, and information from the input device such as the keyboard 180 and the mouse 190 passes through the I / O control unit 200. Are encoded and output from the network connection unit 120 to the information processing apparatus on the other side. The normal TV phone system operates as described above.

  In the present embodiment, the other party's voice acquired from the network connection unit 120 is temporarily stored in the RAM 130, and emotion estimation software for estimating the other party's emotion is started from the ROM 210 and executed by the MPU 110. Then, the other party's emotion is estimated from an emotion database recorded on a disk 220 such as an HDD. The MPU 110 instructs the display 240 to display the auxiliary video (marks such as characters, symbols, and images) based on the estimated emotion together with the video (facial expression) of the other party. As a result, the user can visualize emotions that are difficult to judge from the facial expression of the other person displayed on the display by the auxiliary image, and can easily understand. This is particularly effective when the display is small and it is difficult to judge the emotion from the face of the other party.

  FIG. 2 is a flowchart for estimating a speaker's emotional change. First, the voice data of the speaker transmitted via the network is decoded by the voice recognition system (S110). Next, when talking with a plurality of speaking partners, the speaker is specified from the IP address information (S120). Then, it is estimated whether the speaker's emotion has changed by the following three analysis methods.

The first analysis method uses audio frequency. The voice frequency is analyzed to create a voice frequency distribution (S130). Next, the audio frequency distribution of the past 10 seconds is compared with the unit of seconds (S140). If the voice frequency distribution has changed (S150), it is estimated that the speaker's emotion has changed (S160). On the other hand, if it has not changed, it is estimated that there is no change in the current emotion (S170).

  The second analysis method is performed at the volume level. First, the volume level for the past 10 seconds is compared with the unit of seconds (S180). If the volume level has changed (Yes in S190), it is estimated that the emotion has changed, and the volume level has not changed (S190). No) presumes that there is no change in emotion.

  The third analysis method uses the interval between the own encoding and the decoding request of the other party's story. The data trend of the past 10 seconds at this interval is examined (S200). If the interval tends to decrease or expand (Yes in S210), it is estimated that the emotion has changed, and if there is no such tendency (No in S210), it is estimated that there is no change in emotion. In other words, in a normal conversation, the other party responds immediately after speaking, so a request to decode the other party's story occurs after encoding his voice. However, if the other party is angry, the other party will begin to speak while trying to deny his / her own conversation. In other words, a request to decode the other party's story occurs while decoding one's own voice. On the other hand, when the other party is sad, the reply is delayed because of sadness after speaking. In other words, a request to decode the other party's story occurs after a while after decoding his / her voice.

  FIG. 3 is a flowchart for estimating the emotion of the speaker according to the present embodiment. As an example, a case will be described in which it is estimated whether or not the speaker's emotion corresponds to “anger / excitement” or “sadness”.

  First, the case where the audio frequency distribution has changed to the high sound range (Yes in S310) will be described. In this case, the speaker's emotion may correspond to “anger / excitement”. When it is determined that the volume level is increasing (Yes in S320) and the interval between conversations is decreasing (Yes in S330), the word “detered / excited” recorded in the words and emotion database decoded by the speech recognition system. (S340). If there is a matching word usage (Yes in S350), it can be determined that the emotion of the current speaker is in an “anger / excitement” state. The image display and the symbol corresponding to “anger / excitement” are selected, and the character display font and display color are changed to those that can image “anger / excitement” (S360). If there is no matching word, it is recorded as an additional word in the database (S370).

  On the other hand, if the volume level has not increased (No in S320), or if the talk interval has not decreased (No in S330), the process ends without judging the emotion.

Next, the case where the audio frequency distribution has not changed to the high sound range (No in S310) will be described. In this case, the speaker's emotion may correspond to “sadness”.

  The voice frequency distribution at the start of the conversation is compared with the volume level (S380). When the sound level is changed (Yes in S390), the volume level tends to decrease (Yes in S400), and the interval of the talk is determined to increase (Yes in S410), it is decoded by the voice recognition system. The word is compared with “sadness” recorded in the emotion database (S420). If there is a matching word usage (Yes in S430), it can be determined that the emotion of the current speaker is in the state of “sadness”. The image display and symbol that meet “sadness” are selected, and the character display font and display color are changed to those that can image “sadness” (S440). If there is no matching word, it is recorded as an additional word in the database (S450).

  On the other hand, if the sound level has not changed (No in S380), the volume level has not decreased (No in S390), or the talk interval has not increased (No in S400), the emotion is not judged. finish.

  FIG. 4 is a diagram for explaining the outline of the emotion database recorded on the disc 220. The emotion database is a database based on “joy”, “sadness”, “anger”, and “surprise”. Here, the case of “anger” will be described as an example of a database used for emotion analysis. If there is a corresponding one from the emotion database based on the emotion analysis and the speech recognition result, it is prepared as a display candidate. For example, if the result of speech recognition decoding is “what”, “Nanii!” Is displayed in red.

  5 to 7 are image display examples as a result of emotion analysis. Here, when an angry emotion is analyzed, a background or the like previously selected by the user to be used is displayed. FIG. 5 shows a case where the background is changed, and the background is changed to “a background in which the flame flickers”. 6 and 7 show a case where an image showing emotion is added to the image of the person photographed by the camera. 6 adds a “mark indicating anger” to the forehead, and FIG. 7 adds “boiling kettle” above the head.

  FIG. 8 is a diagram illustrating a display image display example when a conversation is performed by a plurality of people. The analyzed voice is shown in the character display section. In this way, the present embodiment can be applied not only to a one-to-one TV phone but also to a TV chat with three or more people.

  The above-described embodiment is a preferable specific example of the present invention, and thus various technically preferable limitations are attached. However, the embodiments are appropriately combined and changed within a range not departing from the gist of the present invention. It goes without saying that it can be done.

  For example, it is not only the case of the receiving side that analyzes the other party's feelings and displays the other party's feelings on their display. In other words, it is also possible to apply to the case of the transmitting side in which one's own emotion is analyzed, the analysis result is added to his own video, and displayed on the other party's display via the network. As a result, it is possible to convey to the partner a subtle emotional change that is difficult to understand from the expression on the TV screen. Therefore, a hearing-impaired person can more easily understand the emotion of the other party than the conventional TV phone.

  Also, if you have trouble with the language function, enter the words you want to be converted to artificial speech with the keyboard, and assign emotions to the function keys of the keyboard, which is the reverse of the analysis using the emotion database of this embodiment By performing the analysis, it is possible to transmit the voice of the population with emotions to the other party. The other party can add emotional marks to the speaker's facial expression by performing an analysis using the emotion database of this embodiment in addition to the emotional artificial voice. Can express emotions.

1 is a schematic block diagram of an information processing apparatus according to an embodiment of the present invention. The flowchart which estimates the speaker's emotional change which concerns on this embodiment. The flowchart which estimates whether the speaker's emotion which concerns on this embodiment is "anger / excitement" or "sadness". The figure for demonstrating the outline | summary of the emotion database which concerns on this embodiment. The figure which shows the example of an image display of the result of having performed emotion analysis. The figure which shows the example of an image display of the result of having performed emotion analysis. The figure which shows the example of an image display of the result of having performed emotion analysis. The figure which shows the example of a display image display at the time of having conversation by several people.

Explanation of symbols

110 MPU
120 Network connection (ether connection)
130 RAM
140 Camera 150 Image input unit (USB etc.)
160 Microphone 170 Voice input unit 180 Keyboard 190 Mouse 200 I / O control unit 210 ROM
220 Disc 230 Image output unit 240 Display 250 Sound output unit 260 Speaker

Claims (5)

A display for displaying the expression of the speaker;
An analysis unit that analyzes emotions from speaker's voice information;
An information processing apparatus comprising: a control unit that instructs to display the analysis result on the display unit.   The information processing apparatus according to claim 1, wherein the analysis unit analyzes an emotion from a change in frequency of the audio information.   The information processing apparatus according to claim 1, wherein the analysis unit analyzes an emotion from a change in volume of the audio information.   The information processing apparatus according to claim 1, wherein the analysis unit analyzes an emotion from a change in a conversation interval of the voice information.   The information processing apparatus according to claim 1, wherein the analysis result is a mark indicating the emotion of the speaker.

Images