JP2005269207A - Information delivery method and communication apparatus for realizing this method, and program therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To realize more accurate and easier mutual understanding between talkers by enabling an opposite talker to be effectively informed about the sentiment of a talker thereby more smoothing communication. <P>SOLUTION: Each of communication terminals TM1 to TMn judges whether or not a talker makes utterance or responds to an utterer during a video conference communication. When the talker responds to the utterer, The talker's communication terminal detects each of the position of the visual line of the talker and a motion of the head of the talker, estimates the sentiment of the responder on the basis of a result of the detection, and transmits an identification ID representing the estimated sentiment to the communication terminal of the utterer together with a person identification ID. Meanwhile, when the talker makes utterance, the talker's communication terminal receives the sentiment identification ID and the person identification ID of the responder transmitted from the communication terminal of the responder side, selectively reads character data denoting contents of the sentiment corresponding to the received sentiment identification ID from a database 22, superimposes the read character data on received face image data of the corresponding responder and displays the resulting data onto a monitor 8. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an information transmission method for further facilitating communication at the time of a call between conference participants and remote collaborators in a collaborative work support system using, for example, a television telephone device or a video conference system, and The present invention relates to a communication apparatus and a program for realizing this method.

In recent years, various services using communication networks such as the Internet have been proposed. Among them, a remote collaborative work system that supports workers in remote locations using a television telephone device or a video conference system is proposed. Has been. In this type of system, smooth communication between communication partners is extremely important. However, this type of system generally performs communication using only video and audio. For this reason, the intentions and emotions of the caller are not sufficiently transmitted to the call partner, and it is difficult for the speakers to communicate with each other.
In view of this, a television telephone device having a function of transmitting a caller's emotion as an image has been proposed (see, for example, Patent Document 1).

JP 2002-354436 A

  However, in this conventionally proposed television telephone apparatus, only an image representing emotion is transmitted. For this reason, it is difficult to determine whether the other speaker understands what the speaker has spoken or what he wants to convey, or how the other speaker has reacted to the content proposed by the speaker. For example, you cannot immediately confirm whether you are in favor of or against your suggestion, or whether you are pending. For this reason, the communication with the other party speaker is often interrupted, causing a problem that smooth communication cannot be performed. Also, it is difficult to know the emotional state of the other party so that the other party is not aware of it effectively.

  The present invention has been made paying attention to the above circumstances, and the purpose of the present invention is to further facilitate communication by enabling the other speaker to be effectively notified of the speaker's feelings, thereby enabling the speaker An object of the present invention is to provide an information transmission method, a communication device for realizing this method, and a program thereof that make communication between each other more accurate and easy.

  The present invention has been made paying attention to the above circumstances, and the object of the present invention is to perform telephone communication using audio and video between a first communication terminal and a second communication terminal via a communication network. In the first communication terminal, the head movement of the speaker and the position of the line of sight are respectively measured in the first communication terminal, and the emotion of the speaker is estimated based on the measurement results. Information representing the emotion is transmitted to the second communication terminal. Then, the information representing the emotion transmitted from the first communication terminal is presented to the speaker at the second communication terminal.

  Therefore, according to the present invention, the first communication terminal automatically estimates the emotion of the speaker based on the movement of the speaker's head and the position of the line of sight, and the estimated emotion of the speaker is the first communication terminal. 2 is transmitted to the communication terminal 2 and presented to the speaker. For this reason, the speaker of the second communication terminal can grasp the emotion of the other speaker while performing telephone communication. For example, based on the movement of the speaker's head and the position of the line of sight, it is estimated whether the speaker has agreed, denied, or suspended the question, and this is used as information representing the speaker's feelings. If defined, the speaker of the second communication terminal can clearly grasp the other speaker's feelings about his / her question without relying on the other speaker's voice or image. Therefore, communication between speakers during a call can be further facilitated, and communication can be made accurately and easily.

  When transmitting the information representing the heart, the first communication terminal converts the information representing the estimated heart into an identification code, and then transmits the identification code to the second communication terminal. It is preferable to generate and output presentation information representing emotion based on the transmitted identification code. In this way, it is possible to greatly reduce the data amount of information representing emotions, and thus it is possible to transmit information representing emotions easily and in a short time.

Moreover, the following various forms can be considered as a form of presenting information representing emotions.
In the first mode, presentation information that expresses emotions as character data is generated based on the transmitted identification code, and the presentation information that is expressed by the generated character data is transmitted from the first communication terminal. The video information is superimposed on the video information. In this way, the speaker of the second communication terminal can confirm the emotion of the other speaker by using character data while viewing the face image of the other speaker, for example, during videophone communication.

  In the second mode, presentation information that expresses the emotion by voice or acoustic data is generated based on the transmitted identification code, and the presentation information including the generated voice or acoustic data is transmitted from the first communication terminal. It is inserted or synthesized into audio information to be transmitted and output. In this way, the speaker of the second communication terminal can confirm the emotion of the other speaker by means of synthesized speech or acoustic data while listening to the other speaker's voice during telephone communication, for example.

  In the third mode, presentation information that expresses the emotion by air pressure is generated based on the transmitted identification code, and the presentation information that is expressed by the generated air pressure is output. In this way, the speaker of the second communication terminal can check the emotion of the other speaker by the air pressure discharge pattern without changing the received video or the received voice.

  Further, when there are a plurality of first communication terminals, the second communication terminal takes a majority vote of the information representing the emotion transmitted from each of the plurality of first communication terminals, and the result of the majority vote is obtained. It may be presented to the speaker of the second communication terminal. In this way, the speaker of the second communication terminal can quickly determine the feelings of the other call partners from the result of the majority decision in the state where the video conference call is being performed.

In short, in the present invention, when telephone communication using voice and video is performed between a first communication terminal and a second communication terminal via a communication network, the first communication terminal uses the head of the speaker. The movement of the unit and the position of the line of sight are measured, the emotional state of the speaker is estimated based on each measurement result, and information representing the estimated emotional state is transmitted to the second communication terminal. Then, the information representing the emotion transmitted from the first communication terminal is presented to the speaker at the second communication terminal.
Therefore, according to the present invention, it becomes possible to effectively notify the other party's feelings of the speaker, thereby further facilitating communication and making communication between the speakers more accurate and easy. It is possible to provide an information transmission method, a communication device for realizing the method, and a program thereof.

(First embodiment)
FIG. 1 is a block diagram showing a first embodiment of a communication system and a communication apparatus for realizing an information transmission method according to the present invention.
In this communication system, a plurality of communication devices TM1 to TMn are connected to each other via a communication network NW, and video conference communication can be performed between these communication devices TM1 to TMn.

  The communication network NW includes, for example, a wired subscription communication network and a mobile communication network. In addition to PSTN (Public Switched Telephone Network) and ISDN (Integrated Service Digital Network), the wired subscription communication network includes an in-company communication network such as a wired LAN (Local Area Network), and a CATV (Cable Television) network. The mobile communication network includes, for example, a mobile phone system, a PHS (Personal Handyphone System), and a wireless LAN. The communication network NW can include an IP network such as the Internet.

The communication devices TM1 to TMn are configured as follows. Since the communication devices TM1 to TMn have the same configuration, only the communication device TM1 will be described here, and description of the other communication devices TM2 to TMn will be omitted.
The communication device TM1 is a television telephone communication terminal, and functions necessary for television telephone communication include a camera 7, a monitor 8, a microphone 9, a speaker 10, a video control unit 11, an acoustic control unit 12, a multiplexing / separating unit 13, and communication. A processing unit 6 is provided.

  Among these, the video control unit 11 controls the camera 7 and the monitor 8 to capture and encode a face image of a speaker who uses the terminal, and decodes the received video data to monitor 8. To display. The acoustic control unit 12 encodes the transmission voice signal input by the microphone 9, decodes the received voice or acoustic data, and outputs the voice from the speaker 10. The multiplexing / demultiplexing unit 13 multiplexes the video data encoded by the video control unit 11 and the transmission voice data encoded by the acoustic control unit 12 and also generates video from the received multiplexed data. Separate data from voice or acoustic data.

  Further, the communication device TM1 includes a line-of-sight detection unit 1, a head movement detection unit 2, a speaker detection unit 3, a responder processing unit 4, and a speaker processing unit 5A as new functions according to the present invention. I have. As shown in FIG. 2, the line-of-sight detection unit 1 includes an image storage unit 14 and a line-of-sight detection unit 15, and coordinates representing the position of the speaker's line of sight from the speaker's face image data obtained by the video control unit 11. Detect value.

  The head movement detection unit 2 includes a head posture measurement unit 16 and a head movement direction detection unit 17 as shown in FIG. The head posture measuring means 16 is constituted by a three-axis posture angle sensor such as a gyroscope or a magnetic sensor, and is mounted on the head of the speaker. The head movement direction detection means 17 detects the movement direction (angle) of the speaker's head based on the detection signal output from the head posture measurement means 16.

The speaker detection unit 3 includes a speaker's line-of-sight position and head movement angle detected by the line-of-sight detection unit 1 and the head movement detection unit 2, and transmission voice data output from the acoustic control unit 12. Based on the above, it is determined whether or not the speaker is speaking.
When the speaker detection unit 3 determines that the speaker is not speaking, that is, when the speaker processing unit 4 determines that the speaker is currently receiving (in response), the responder processing unit 4 2) and is configured as shown in FIG. That is, the responder processing unit 4 includes a feeling identification unit 18 and a feeling identification result storage unit 19. The emotion identification means 18 is based on the speaker's line-of-sight position and head movement angle detected by the line-of-sight detection unit 1 and head movement detection unit 2, and the emotional information of the speaker who is receiving (responding). The process which estimates is performed. Then, the estimated emotion information is converted into a emotion identification ID representing the type of emotion. The sentiment identification result storage means 19 stores the sentiment identification ID obtained by the sentiment identification means 18 in the memory in association with the identification ID of the speaker who is receiving (responding).

  When the speaker detection unit 3 determines that the speaker is speaking, the speaker processing unit 5A displays the other speaker's emotional information about the content of the speaker's speech to the communication terminals TM2 to TMn of the partner speaker. And presenting it to the speaker who is speaking, as shown in FIG. 3, a database (DB) 22, a reception result storage means 24, a database search means 23, a search result storage means 21, Superimposing means 20.

  As shown in FIG. 4, the database 22 includes a person identification table 25 and a feeling identification table 29. In the person identification table 25, the participant (speaker) name 27 and the emotion identification ID 28 are stored in association with the person identification ID 26, respectively. The emotion identification table 29 stores a text file name 30 expressing the emotion in association with the emotion identification ID 28.

  The reception result storage means 24 stores and holds the received emotion identification ID and speaker identification ID. The database search means 23 searches the database 22 based on the received emotion identification ID and person identification ID, and reads character data expressing the corresponding emotion. The search result storage means 21 stores and holds character data representing the read emotion. The superimposing means 20 performs a process for superimposing the character data expressing the emotion on the video data separated by the multiplexing / separating unit 13.

  The communication processing unit 6 transmits / receives the multiplexed data, the emotion identification ID, and the speaker identification ID to / from the communication terminals TM1 to TMn as communication partners according to the communication protocol defined by the communication network NW. As the communication protocol, TCP / IP (Transmission Control Protocol / Internet Protocol) is used if the communication network NW includes, for example, an IP network.

Next, the operation of the communication terminal TM1 configured as described above will be described.
When the television conference communication is started, the communication terminal TM1 transmits and receives video and audio, and estimates the emotional information of the speaker and performs transmission / reception processing thereof. FIG. 5 is a flowchart showing the procedure and contents of emotion information estimation and transmission / reception processing.

  That is, the communication terminal TM1 first detects the line-of-sight position of the call participant in step 5a. As a method for detecting the line-of-sight position, for example, the technique described in “Gaze Measurement Method Based on Eyeball Shape Model: Image Sensing Symposium, 2002” can be used. Next, in step 5b, the three-axis posture angle of the movement of the speaker's head is detected based on the detection signal of the gyroscope or magnetic sensor attached to the call participant's head. In step 5c, it is determined based on the audio signal output from the acoustic control unit 12 whether the speaker is speaking or receiving (responding).

As a result of the determination, if the speaker is responding, the process proceeds to step 5g, where the responder processing unit 4 determines the emotion information of the responder, and in step 5h, the speaker who is speaking is equivalent to this emotion information. A sentiment identification ID and a person identification ID are transmitted. On the other hand, when the speaker is speaking, the process proceeds to step 5d to receive the emotional information ID and person identification ID sent from the communication terminals TM2 to TMn that are receiving (responding). Then, the speaker processing unit 5A selects a presentation method of emotion information based on the line-of-sight information included in the responder's video (step 5e), and the received responder according to the selected presentation method. Is presented (step 5f).
The above processing is continued until the video conference communication ends.

  By the way, the line-of-sight position detection process of the call participant by the line-of-sight detection unit 1, the head movement detection process of the speaker participant by the head movement detection unit 2, and the speech detection by the speaker detection unit 3 is in response. For example, the determination process of the presence and the estimation process of the emotion information by the responder processing unit 4 are performed as follows, for example. FIG. 6 is a flowchart showing the processing procedure and processing contents.

  First, in step 6a, the face image of the speaker captured by the camera 7 is captured from the video control unit 11, and in step 6b, the line-of-sight position is detected by performing image processing on the captured face image. This line-of-sight position is detected by acquiring the coordinates of the eyeball in the screen of the monitor 8. This process is performed for each image frame rate. When the coordinate position is detected, the process proceeds from step 6c to step 6d, where the difference in the line-of-sight position coordinates between the frames is calculated, and the movement amount of the line-of-sight position is calculated. Then, by using the calculated movement amount of the line-of-sight position, it is detected whether or not the line of sight is in a gaze state in step 6e and step 6f.

  Specifically, first, the amount of line-of-sight movement between consecutive frames is calculated. If the calculated line-of-sight movement amount is within a preset threshold value, the movement amount of the line-of-sight position from the next frame onward is calculated on the basis of the line-of-sight position of the difference source frame. If the calculated movement amount of the line-of-sight position is continuously within a preset threshold value and the number of continuous times within the threshold value is greater than or equal to the preset number of times, that is, greater than or equal to the set time, The state of time is considered to be gazing at the position where the line of sight is.

  Next, in parallel with the eye gaze position detection process, the posture angle of head movement is measured. Now, suppose that head movements are defined as three types of movements of “neck wagging”, “neck wagging”, and “nodding”, and the angles corresponding to these movements are (α, β, γ). First, in step 6g, the three-axis posture angle (α, β, γ) of the head is measured for each sampling period, and the amount of change is calculated for each posture angle in step 6h. Then, the head movement is discriminated as follows based on the calculated change amount.

  For example, with regard to neck swing (posture angle: β) and nodding (posture angle: γ), the amount of change in the posture angle is first set to a preset value or more in step 6i and step 6j. Judge whether it is above. If this condition is satisfied, it is subsequently determined in step 6l whether there is a sign change in the continuous change amount. This is because “neck swing” and “nodding” are considered to be continuous reciprocation of the head. If there is a sign change as a result of this determination, it is regarded as a reciprocal operation and the number of times is stored. If the number of times is greater than or equal to the setting in step 6m, it is considered that “neck shake” and “nodding” have been performed. On the other hand, since the neck curl (posture angle: α) is not a reciprocal motion, if the amount of change per sampling period is greater than or equal to the set value and the latest posture angle is greater than or equal to the set value in step 6k, the neck is curled. It is considered to be.

  After detecting the head movement according to the above procedure, the emotion is determined by combining with the gaze state of the line-of-sight position detected first. For example, when each action and gaze is performed continuously, if the posture angle is γ (nod), it is determined to agree, and if the posture angle is β (fluff), it is determined to be negative. ) Is determined to be on hold. In step 6o, step 6p, and step 6q, an identification ID corresponding to each of the determined emotions is given, and the emotion identification ID is transmitted to the communication terminal of the speaker who is speaking together with the person identification ID.

On the other hand, the process of presenting the emotion identification ID and the person identification ID sent from the responder's communication terminal is performed as follows. FIG. 7 is a flowchart showing the emotion information presentation processing procedure and processing contents by the speaker processing unit 5A.
When person identification IDs and emotional identification IDs arrive from a plurality of communication terminals on the responder side in response to the utterance of the speaker, these personal identification IDs and emotional identification IDs are received by the communication processing unit 6 under the control of step 7a. After that, it is taken into the speaker processing unit 5A and stored in the reception result storage means 24 in step 7b.

  When the person identification ID and the emotion identification ID are received and stored, the speaker processing unit 5A searches the database 22 based on these identification IDs in step 7c. Then, a corresponding speaker is selected from participating speakers based on the person identification ID (step 7d), and a text file name representing emotion information for each corresponding speaker is selected based on the emotion identification ID. Then, the file is acquired (step 7e). The face image of the corresponding speaker displayed on the monitor 8 is identified based on the person identification ID (step 7f), and the character data in the text file is superimposed on the face image of the corresponding speaker. (Step 7g). The face image of the responder on which the character data is superimposed is supplied to the monitor 8 via the video control unit 11 and displayed.

  FIG. 8 shows a display example of the respondent's face image on which the character data is superimposed. As shown in the figure, the main responder's face image and character data representing its emotion are displayed on the main screen 32 of the monitor 8, and each of the other respondent's face images and their emotions are displayed on the sub-screens 32-34. Character data representing is displayed.

  As described above, in the first embodiment, each of the communication terminals TM1 to TMn determines whether the speaker is speaking or responding during the video conference communication. When a response is being made, the speaker's gaze position and head movement are detected, the responder's emotion is estimated based on the detection results, and an identification ID representing the estimated emotion is assigned to the person. It is transmitted to the communication terminal of the speaker together with the identification ID. On the other hand, when speaking, the respondent's emotional identification ID and person identification ID sent from the responder's communication terminal are received, and character data representing the emotional content corresponding to the received emotional identification ID is stored in the database. 22 is selectively read out and the read character data is superimposed on the received face image data of the corresponding responder and displayed on the monitor 8.

  Therefore, the communication terminals TM1 to TMn automatically estimate the emotion of the responding speaker. The estimated emotion of the speaker is transmitted to the communication terminal on the speaker side and converted into character data. Are displayed together with the face image. For this reason, the speaker who is speaking can easily and accurately grasp the responder's feelings about his / her speech from the character data while performing the television telephone communication. For example, it is possible to clearly grasp whether the responder has agreed, denied, or suspended the inquiry without relying on guesses from the responder's voice or images. Therefore, communication between speakers during a call can be further facilitated, and communication can be made more accurate and easy.

(Second Embodiment)
FIG. 9 is a block diagram showing a main configuration of a communication terminal according to the second embodiment of the present invention. In the figure, the same parts as those in FIGS. 1 to 3 are denoted by the same reference numerals, and detailed description thereof is omitted.
The speaker processing unit 5B of the communication terminal according to the second embodiment includes a processing unit that converts emotion information into character data and presents it, and a process that converts the emotion information of the responder into media other than character data and presents it. It consists of a part.

  As in FIG. 3, the processing unit for converting to character data and presenting the received result storage means 24 for storing the received person identification ID and emotion identification ID from the plurality of responder terminals, Based on each identification ID, the database search means 22 for searching the character data representing the corresponding feeling from the database 22, the search result storage means 21 for storing the searched character data, and the searched character data as the responder Superimposing means 20 for superimposing on the received face information.

  On the other hand, the processing unit that converts the emotion information to other media and presents it includes the superimposed image storage means 43, the feature area extraction means 44, the presentation technique identification means 36, the database (DB2) 40, and the database search means 39. Voice / acoustic file selection means 37, voice / sound output means 38, air pressure control means 41, and air pressure generation means 42.

  The superimposed image storage unit 43 stores the face image data of the responder on which the character data is superimposed in the superimposing unit 20. The feature area extraction unit 44 extracts the gaze position of the speaker detected by the gaze detection unit 1 and the feature area of the superimposed image. The presentation method identification means 36 determines what kind of media the responder's emotional information is converted to be presented.

  The database 40 stores other media data representing emotion information, and includes a person identification table 25, a emotion identification table 29, and a line-of-sight position table 46 as shown in FIG. The person identification table 25 stores the participant (speaker) name 27 and the emotion identification ID 28 in association with the person identification ID 26. The emotion identification table 29 stores a text file name 30 expressing the emotion and a line-of-sight position ID in association with the emotion identification ID 28. In the line-of-sight position table 46, a voice / acoustic file name 47 expressing a feeling and an air pressure parameter setting file name 48 are stored in association with the line-of-sight position ID.

  The database search means 39 searches the database 40 based on the received emotion identification ID and person identification ID and the detected line-of-sight position ID based on the determination result of the presentation technique determination means 36. The voice / acoustic file name 47 or the pneumatic parameter setting file name 48 expressing the corresponding emotion is read out.

  The voice / sound file selection unit 37 corresponds to the voice / sound file name read by the database search unit 39 when the presentation method using the voice / sound data is designated by the presentation method determination unit 36. Select a voice / acoustic file. The voice / sound output unit 38 outputs the selected voice / sound file from the speaker 10.

  The air pressure control means 41 selects a discharge pattern of air pressure corresponding to the air pressure parameter setting file name read out by the database search means 39 when the presentation technique using the air pressure is designated by the presentation technique discriminating means 36. To do. The air pressure generating means 42 discharges air pressure according to the selected air pressure discharge pattern.

Next, the emotion information presentation processing operation by the speaker processing unit 5B of the communication terminal configured as described above will be described. FIG. 11 is a flowchart showing the processing procedure and processing contents.
When face image data of the speaker is sent from the communication terminals of a plurality of responders, the face image data is received by the communication processing unit 6, the multiplexing / separating unit 13 and the video control unit 11 under the control of step 11a. After that, it is stored in the reception result storage means in step 11b.

  At the same time, when the person identification ID and the emotion identification ID are sent from each responder's communication terminal, these identification IDs are received by the communication processing unit 6 under the control of step 11c and then stored in the reception result storage unit 24. Once memorized. First, in step 11d, the speaker processing unit 5B searches the database 40 based on each received identification ID, reads the text file name matching the emotion identification ID, and is described in this text file. Get character data. In step 11f, the acquired character data is superimposed on the received responder's face image data, and the face image data on which the character data is superimposed is displayed on the monitor 8 in step 11g. Further, the face image data superimposed with the character data is stored in step 11h, and the difference from the received face image is calculated in step 11i. Then, based on the calculated difference, a feature region is extracted in step 11j.

  On the other hand, in step 11k, the speaker processing unit 5B captures the face image data of the speaker captured by the camera 7 from the video control unit 11 to the line-of-sight detection unit 1, and in step 11l, the speaker face image data captured in step 11l. To calculate the line-of-sight position coordinates (x, y) in the screen. Then, based on the calculated line-of-sight position coordinates and the feature area extracted in step 11j, it is identified in step 11m which of the character data superimposition area, the person area, and other areas the line-of-sight position exists. The line-of-sight position ID corresponding to the identified area is acquired.

  Next, the speaker processing unit 5B executes emotion information presentation processing according to the identification result. For example, if the line-of-sight position is in the person area, the database 40 is searched based on the line-of-sight position ID in step 11n to obtain the corresponding voice / acoustic file name, and the acquired voice / acoustic file name is used. The voice / sound file represented is output from the speaker 10 under the control of step 11o. Therefore, in this case, the responder's emotional information is presented to the speaker by, for example, synthesized voice or sound.

  On the other hand, when the line-of-sight position is in the overlapping area with the character data, the speaker processing unit 5B searches the database 40 based on the line-of-sight position ID in the same manner as in the case of the person area in step 11p. The corresponding voice / acoustic file name is acquired, and the voice / acoustic file represented by the acquired voice / acoustic file name is output from the speaker 10 under the control of step 11q. At the same time, in step 11r, the database 40 is searched based on the visual field position ID, and a file name in which air pressure parameters such as the magnitude of discharge pressure and discharge time for generating air pressure are described is obtained. Based on the parameters described in the acquired file, air pressure and pressure discharge time are set in steps 11s and 11t, respectively, and air pressure is generated in step 11u according to the set air pressure and pressure discharge time. Air is discharged from the means 42 to the speaker who is speaking.

  If the line-of-sight position is neither a person area nor a character data superimposition area, the speaker processing unit 5B acquires the air pressure parameter setting file name from the database 40 based on the line-of-sight position ID in step 11v. Then, according to the contents of the acquired air pressure parameter setting file, the air pressure and the pressure discharge time are respectively set at step 11w and step 11x, and from the air pressure generating means 42 at step 11y according to the set air pressure and pressure discharge time. Air is discharged toward the speaker who is speaking.

As described above, in the second embodiment, when presenting the respondent's emotional information, the line-of-sight position is detected based on the face image of the speaker, and the detected line-of-sight position Determine if it is in position. If it is determined that the visual field position is in the person area, the emotional information is converted into voice / acoustic information and output. If it is determined that the line-of-sight position is in the superimposed area with the character data, the emotional information is output. Information is converted into voice / acoustic information and output, and also converted into a pneumatic pattern to discharge air. If the line-of-sight position is neither a person area nor a character data superimposition area, the emotion information is converted into an air pressure pattern and air is discharged.
Therefore, the respondent's emotional information is presented by the optimal presentation method according to the visual field position of the speaker, so that the speaker can further improve the responder's emotion to his / her speech while performing a video conference communication. It becomes possible to grasp accurately.

(Third embodiment)
FIG. 12 is a block diagram showing a main configuration of a communication terminal according to the third embodiment of the present invention. In the figure, the same parts as those in FIGS. 1 to 3 and FIG.
The speaker processing unit 5C of the communication terminal according to the third embodiment includes a processing unit that converts emotion information into character data and presents, a processing unit that presents the number of responders for each emotion information, And a processing unit for presenting emotional information that maximizes the number of respondents.

  The processing unit that converts to character data and presents the received result storage means 24 that stores the received person identification IDs and emotional identification IDs from the plurality of responder terminals, as in FIGS. Based on each received identification ID, database search means 22 for searching for character data representing the corresponding emotion from the database 22, search result storage means 21 for storing the searched character data, and the searched character data Is superimposed on the received face information of the responder.

  The processing unit that presents the number of responders by emotion information includes a video storage unit 49, a region detection unit 50, a number-of-hearts output unit 51, and a number-of-persons calculation unit 52 by emotion information. The video storage unit 49 stores face image data captured by the camera 7. The area detecting means 50 detects an area of a character information display window that displays character data. The number calculation means 52 according to the emotion information counts the number of responders according to the feeling based on the feeling identification ID sent from the communication terminal of each responder. The number-of-heart-indicating number display means 51 displays the calculated number of responders according to the number of feelings on the monitor 8.

  The processing unit that presents the emotion information that maximizes the number of responders for each emotion information includes a determination unit 53, a voice / acoustic file selection unit 37, a voice / acoustic output unit 38, an air pressure setting file selection unit 54, Air pressure control means 41 and air pressure generation means 42 are provided. The determination means 53 determines the emotion information that is the maximum number based on the number of responders according to emotion. The voice / sound file selection unit 37 and the voice / sound output unit 38 present voice / sound data corresponding to the emotion of the maximum number of people to the speaker based on the determination result by the determination unit 53. The air pressure setting file selection means 54, the air pressure control means 41, and the air pressure generation means 42 present the emotion of the maximum number of people to the speaker by air pressure based on the determination result by the determination means 53.

Next, the emotion information presentation processing operation by the speaker processing unit 5C of the communication terminal configured as described above will be described. FIG. 13 is a flowchart showing the processing procedure and processing contents.
When a person identification ID and a feeling identification ID arrive from a plurality of communication terminals on the responder side with respect to the utterance of the speaker, the person identification ID and the feeling identification ID are received by the communication processing section 6 and then the speaker processing section 5C and temporarily stored in the reception result storage unit 24.

  When the person identification ID and the emotion identification ID are received and stored, the speaker processing unit 5C searches the database 22 based on these identification IDs in step 13a. Then, a corresponding speaker is selected from participating speakers based on the person identification ID, and a text file name representing emotion information for each corresponding speaker is selected based on the emotion identification ID. Get the file. Further, the face image of the corresponding speaker displayed on the monitor 8 is identified based on the person identification ID, and the superimposing means 20 superimposes the character data in the text file on the corresponding face image of the speaker. The face image of the responder on which the character data is superimposed is supplied to the monitor 8 via the video control unit 11 and displayed.

  Next, under the control of step 13c, the speaker processing unit 5C calculates the number of responders according to the emotion information based on the emotion identification ID by the number-of-hearts calculation means 52. At the same time, the line-of-sight position on the monitor screen of the speaker is detected in step 13c, and it is determined in step 13d whether the detected line-of-sight position is in the character information display window area. When it is determined that the line-of-sight position is in the character information display window area, the process proceeds to step 13e, where the number of responders according to the emotion information calculated by the number of persons according to the emotion is displayed. It is displayed on the monitor 8 by means 51. On the other hand, when it is determined that the line of sight of the speaker is not in the character information display window area, the process of detecting and determining the line of sight in steps 13c and 13d is repeated.

  FIG. 14 shows a display example of the number of responders according to their emotional information. As shown in the figure, a character information display window area 55 is provided on the main screen 32. In this character information display window area 55, the number of responders according to emotion information, that is, “agree”, “disagree”, and “ Each number of “pending” is displayed.

  Further, the speaker processing unit 5C selects the emotional information that maximizes the number of responders based on the number of emotional information calculated by the emotional number-of-people calculating means 52 in step 13f. Then, the voice / acoustic file and the air pressure parameter setting file corresponding to the emotional information that maximizes the number of selected responders are acquired from the database 22, respectively. Then, based on the acquired sound / sound file, the sound / sound file is output from the speaker 10 in step 13g. At the same time, the air pressure corresponding to the discharge force parameter and the discharge time parameter described in the acquired air pressure parameter setting file is discharged from the air pressure generating means 42 in step 13h. As a result, the speaker can immediately grasp the rough feelings of all responders to his / her speech.

As described above, in the third embodiment, when presenting respondent's emotion information, the number of responders according to the emotion information is calculated and displayed in the character information display window area 55 of the monitor 8. At the same time, based on the calculated number of responders for each emotional information, the emotional information that maximizes the number of respondents is selected, and the selected emotional information is synthesized using synthesized speech and air pressure. They are presented to the speaker.
Therefore, during the video conference communication, the speaker can grasp the emotional information of each responder to his / her utterance individually from the character data, and further maximize the number of responders by number of emotional information. It becomes possible to grasp at a glance the emotion information, that is, the majority of respondent's emotions.

(Other embodiments)
When displaying the character data representing the responder's feelings, the gaze position of the speaker is detected, and the display size of the character data representing the responder's feeling corresponding to the detected gaze position is determined by the other responders. It may be configured to be larger than the display size of the character data representing the emotion. If it does in this way, it will become possible to visually recognize more clearly the character data showing the respondent's feelings which the speaker is gazing at.

  As a transmission method of emotion information, a method of transmitting emotion information as a part of control data separately from video data and audio data, or character data representing emotion information is multiplexed and transmitted together with video data and audio data in a common packet. It is possible to use a method to do so. At this time, as a multiplexing method, for example, a method used for multimedia multiplex transmission such as a method defined in ITU-T H.223 can be applied.

  In the above embodiment, the case where the video conference communication is performed between three or more talks has been described as an example. However, the video phone communication between the two parties or the communication between the two parties only by voice is performed. Even in this case, the present invention can be applied. Furthermore, as a communication terminal used at that time, it is possible to use a mobile communication terminal such as a PDA (Personal Digital Assistants) or a mobile phone in addition to a wired communication terminal such as a personal computer.

Other types of emotional information and detection methods, presentation forms, emotional information transmission methods, types and configurations of communication terminals, types and configurations of communication networks, communication protocols, and the like do not depart from the spirit of the present invention. Various modifications can be made.
In short, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in each embodiment. Furthermore, you may combine suitably the component covering different embodiment.

The block diagram which shows the structure of the communication system and communication terminal concerning 1st Embodiment of this invention. The block diagram which shows the structure of the responder process part of the communication terminal shown in FIG. 1, and its peripheral part. The block diagram which shows the structure of the speaker process part of the communication terminal shown in FIG. 1, and its periphery part. The figure which shows the structure of the database used with the speech process part shown in FIG. The flowchart which shows the transmission / reception processing procedure and processing content of heart information by the communication terminal shown in FIG. FIG. 6 is a flowchart showing the emotion information estimation processing procedure and processing contents in the emotion information transmission / reception processing procedure shown in FIG. 5; FIG. 6 is a flowchart showing the emotion information reception presentation processing procedure and processing contents in the emotion information transmission / reception processing procedure shown in FIG. 5; The figure which shows an example of the display result of heart information. The block diagram which shows the principal part structure of the communication terminal concerning the 2nd Embodiment of this invention. The figure which shows the structure of the database used by the speech process part in the communication terminal shown in FIG. The flowchart which shows the reception presentation process procedure of the emotion information by the communication terminal shown in FIG. 9, and the processing content. The block diagram which shows the principal part structure of the communication terminal concerning the 3rd Embodiment of this invention. The flowchart which shows the presentation process procedure and processing content of the heart information by the communication terminal shown in FIG. The figure which shows an example of the display result of the feeling information by the feeling information presentation process shown in FIG.

Explanation of symbols

  TM1 to TMn: communication terminal, NW: communication network, 1 ... gaze detection unit, 2 ... head motion detection unit, 3 ... speaker detection unit, 4 ... responder processing unit, 5A, 5B, 5C ... speaker processing unit , 6 ... Communication processing unit, 7 ... Camera, 8 ... Monitor, 9 ... Microphone, 10 ... Speaker, 11 ... Video control unit, 12 ... Sound control unit, 13 ... Multiplexing / separating unit, 14 ... Image storage means, 15 ... Line of sight detection means, 16 ... head posture detection means, 17 ... head movement direction detection means, 20 ... superimposition means, 21 ... search result storage means, 22 ... database (DB), 23 ... database search means, 24 ... reception results Storage means, 25 ... person identification table, 26 ... person identification ID, 27 ... person identification ID, 28 ... emotion identification ID, 29 ... emotion identification table, 30 ... text file name, 31 ... emotion information presentation screen, 32-35 ... Tatami screen, 36 ... presentation method discriminating means, 37 ... voice / sound file selecting means, 38 ... sound / sound output means, 39 ... database search means, 40 ... database (DB2), 41 ... pneumatic control means, 42 ... pressure generation Means 43 ... Superimposed image storage means 44 ... Feature region extraction means 45 ... Eye position ID 46 ... Eye position table 47 ... Sound / acoustic file name 48 ... Air pressure parameter setting file name 49 ... Video storage means 50 ... area detection means, 51 ... number-of-hearts display means, 52 ... number-of-hearts calculation means, 53 ... determination means, 54 ... air pressure setting file selection means, 55 ... character information display window.

Claims (19)

When performing telephone communication using audio and video via a communication network between the first communication terminal and the second communication terminal,
Measuring the movement of the speaker's head in the first communication terminal;
Measuring the position of the speaker's line of sight in the first communication terminal;
In the first communication terminal, a process of estimating the emotion of the speaker based on the measurement result of the head movement and the measurement result of the position of the line of sight;
Transmitting information representing the estimated emotion from the first communication terminal to the second communication terminal;
The information communication method comprising: a step of presenting information representing the transmitted emotion to a speaker of the second communication terminal in the second communication terminal. In the process of transmitting the information representing the emotion, the information representing the estimated emotion is converted into an identification code, and the converted identification code is transmitted from the first communication terminal to the second communication terminal.
The step of presenting the transmitted information representing the emotion to the speaker generates presentation information representing the emotion based on the transmitted identification code, and outputs the generated presentation information. The information transmission method according to claim 1.   The process of presenting the transmitted information representing the emotion to the speaker generates presentation information representing the emotion in character data based on the transmitted identification code, and is represented by the generated character data. 3. The information transmission method according to claim 2, wherein the presentation information is displayed superimposed on video information transmitted from the first communication terminal.   The process of presenting the transmitted information representing the emotion to the speaker generates presentation information representing the emotion by voice or acoustic data based on the transmitted identification code, and the generated voice or acoustic data. 3. The information transmission method according to claim 2, wherein the presentation information consisting of: is inserted into or synthesized with voice information transmitted from the first communication terminal.   The process of presenting the transmitted information representing the emotion to the speaker generates presentation information representing the emotion in air pressure based on the transmitted identification code, and presenting information represented by the generated air pressure. The information transmission method according to claim 2, wherein: When there are a plurality of the first communication terminals,
In the second communication terminal, a process of taking a majority vote of information representing the emotion transmitted from each of the plurality of first communication terminals and presenting a result of the majority vote to a speaker of the second communication terminal, The information transmission method according to claim 1, further comprising: A communication device that performs telephone communication using audio and video via a communication network with a communication device on the other side,
Means for measuring the movement of the speaker's head;
Means for measuring the position of the speaker's line of sight;
Means for estimating the emotion of the speaker based on the measurement result of the movement of the head and the measurement result of the position of the line of sight;
A communication apparatus comprising: a transmission unit configured to transmit information representing the estimated emotion to the communication apparatus on the other side.   8. The communication apparatus according to claim 7, wherein the transmission means converts information representing the estimated emotion into an identification code, and transmits the converted identification code to the communication apparatus on the other side. A communication network with a communication device on the other side having a function of measuring the movement of the speaker's head and the position of the line of sight and estimating and transmitting the emotion of the speaker based on each measurement result A communication device for performing telephone communication using audio and video via
Receiving means for receiving information representing the sentiment transmitted from the communication device on the other side;
A communication device comprising: presentation means for presenting the information representing the received emotion to a speaker using the device. When the communication device on the other side has a function of converting the information representing the estimated emotion into an identification code and transmitting the converted identification code,
The receiving means receives an identification code transmitted from the counterpart communication device,
The communication device according to claim 9, wherein the presenting unit generates presenting information representing emotion based on the received identification code, and outputs the generated presenting information.   The presenting means generates presentation information representing emotions as character data based on the received identification code, and superimposes the presentation information represented by the generated character data on the received video information. The communication device according to claim 10, wherein the communication device is displayed.   The presenting means generates presentation information that expresses the emotion by voice or acoustic data based on the received identification code, and the presentation information represented by the generated voice or acoustic data is received by the received voice. 11. The communication apparatus according to claim 10, wherein the communication apparatus outputs the information by inserting or combining the information.     11. The presenting means generates presentation information that expresses a heart by air pressure based on the received identification code, and outputs the presentation information represented by the generated air pressure. Communication equipment. When there are a plurality of communication devices on the other side,
The receiving means receives information representing the sentiments respectively transmitted from the plurality of counterpart communication devices,
The communication device according to claim 9, wherein the presenting unit further includes a function of taking a majority vote of the information representing the plurality of received emotions and presenting a result of the majority vote to a speaker. A program used in a communication device including a computer for performing telephone communication using audio and video via a communication network with a communication device on the other side,
Processing to detect the movement of the speaker's head;
Detecting the position of the speaker's line of sight;
A process of estimating the emotion of the speaker based on the detection result of the movement of the head and the detection result of the position of the line of sight;
A program for causing the computer to execute a process of transmitting information representing the estimated emotion to the communication apparatus on the other side. The process to send is
A process of converting information representing the estimated emotion into an identification code;
16. The program according to claim 15, which causes the computer to execute a process of transmitting the converted identification code to the communication apparatus on the other side. A communication network with a communication device on the other side having a function of measuring the movement of the speaker's head and the position of the line of sight and estimating and transmitting the emotion of the speaker based on each measurement result A program used in a communication device including a computer for performing telephone communication using audio and video via
A process of receiving information representing the sentiment transmitted from the communication device on the other side;
A program for causing the computer to execute a process of presenting the information representing the received emotion to a speaker who uses the apparatus. When the communication device on the other side has a function of converting the information representing the estimated emotion into an identification code and transmitting the converted identification code,
The receiving process receives an identification code transmitted from the counterpart communication device,
18. The program according to claim 17, wherein the presenting process generates presentation information representing a feeling based on the received identification code, and outputs the generated presentation information. When there are a plurality of communication devices on the other side,
The receiving process receives information representing a sentiment transmitted from the plurality of counterpart communication devices, respectively.
The program according to claim 17, wherein the presenting process further includes a processing function of taking a majority vote of the received information representing a plurality of emotions and presenting a result of the majority vote to a speaker.

Images