JP2015220534A - Auxiliary apparatus, auxiliary system and auxiliary method for communication, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique which can adjust the degree of presenting the existence of the self to an opposite party.SOLUTION: There is provided a communication auxiliary apparatus for assisting communication between a video conference terminal 10A disposed in a first base 11A and a video conference terminal 10B disposed in a second base 11B through a communication network N. The first base includes: a wireless communication unit 53 for receiving a second base state information which represents an estimated state of the second base; a sense quantity range calculation unit 55a for calculating the range of a sense amount which a first participant 13 located in the first base is perceptible; an intention amount calculation unit 55b for calculating the intention amount of a second participant located in the second base, on the basis of the second base state information and the range of the sense amount of the first participant; and a stimulus output unit 57 for outputting a stimulus according to the intention amount so that can be perceived by the first participant.

Description

  The present invention relates to a communication assistance device, a communication assistance system, a communication assistance method, and a program suitable for assisting communication between remote locations via a communication network.

In recent years, communication network technology has improved and communication with remote locations can be easily performed. For example, in teleconferences held between remote locations, by transmitting and receiving audio and video, the conference can be carried out without moving the base, so there is little space restriction and demand is increasing. It is used in the scene.
In addition, it is possible to easily communicate between remote locations by sending and receiving characters such as e-mail and chat on mobile phones.
However, in remote communication using voice, video, and characters, participants do not actually share one space.
For example, in a video conference, when a video conference is heated in one base, there is a case where the presence of a participant in a different base may be forgotten, and a problem that the conference is excited in one base is likely to occur.
For example, in a video conference, the situation of different sites at one site is displayed as a video on the monitor, but since the video is limited, video information (non-linguistic information) that can be sufficiently visually confirmed can be transmitted. Such a problem is likely to occur.

In Patent Document 1, as an example of performing a conference by adding non-linguistic information to a video, a participant candidate who is going to speak next is based on information indicating a degree of synchronization with the speaker. Selected and presented to each participant.
Further, in Non-Patent Document 1, a robot having a moving function and a camera video shooting function allows a remote participant to control the robot and determine where he (participant) is looking at. To present. Furthermore, the presence is expressed by giving the participants of different bases the awareness of being seen.

Patent Document 1 is effective only when a participant attempts to speak. On the other hand, it is not effective when the participant wants to be aware of his / her presence by a participant at a different base, not until the participant speaks. In other words, there is a problem that the degree of expressing one's presence that he wants to present to participants at different bases cannot be adjusted.
In addition, when only materials are displayed on a monitor in a video conference, the participants are not displayed on the monitor, so there is a problem that participants in different locations may not care if they want to be aware of their existence. there were.
Further, it is difficult for Non-Patent Document 1 to present a presence casually. When the robot moves around during the video conference, the stimulus given to the participant's sense is strong, and the presence of the robot is presented more than necessary. Therefore, there is a problem that it is impossible to adjust the degree of existence that one wants to present to participants at different bases.
As described above, in the conventional remote communication, there is a problem that the amount representing the degree of presenting the presence of the user to the other party cannot be adjusted, and the presence of the participants in the remote place is lowered. On the other hand, when using a robot, there was a problem that participants at remote locations were strongly presented.
The present invention has been made in view of the above, and an object thereof is to provide a technique capable of adjusting the degree of presenting his / her presence to the other party.

  In order to solve the above problem, the invention according to claim 1 is a communication for assisting communication via a network between the first conference terminal arranged at the first base and the second conference terminal arranged at the second base. An auxiliary device, wherein the first site is perceivable by a receiving means for receiving second site status information representing the estimated status of the second site and a first participant at the first site Based on the sensory amount range calculating means for calculating the sensory amount range, the second site status information and the sensory amount range of the first participant, the intention amount of the second participant at the second site is calculated. Intention amount calculating means, and stimulus output means for outputting the stimulus corresponding to the intention amount so that the first participant can perceive.

  According to the present invention, the second participant intention amount at the second base is calculated based on the second site status information and the range of the first participant's sense amount, and the first stimulus corresponding to the intention amount is calculated. By outputting so that a participant can perceive, it is possible to adjust the degree of presenting his / her presence to the other party.

It is a schematic diagram for demonstrating the structure by which the communication assistance system which concerns on 1st Embodiment of this invention is used for the video conference system. It is a block diagram which shows the communication assistance system which concerns on 1st Embodiment of this invention. It is a flowchart showing operation | movement of the communication assistance system which concerns on 1st Embodiment of this invention. It is a flowchart showing operation | movement of the communication assistance system which concerns on 2nd Embodiment of this invention, and is especially a flowchart which shows the specific example of step S200 shown in FIG. It is a flowchart showing operation | movement of the communication assistance system which concerns on 3rd Embodiment of this invention, and is a flowchart showing the specific example of the 1st location situation calculation process (step S210) shown in FIG. 4 especially. It is a flowchart showing operation | movement of the communication assistance system which concerns on 4th Embodiment of this invention, and is especially a flowchart for demonstrating the specific example of the 2nd site | part condition calculation process (step S250) shown in FIG. It is a flowchart showing operation | movement of the communication assistance system which concerns on 5th Embodiment of this invention, and the flowchart explaining the specific example of the 1st site | part condition calculation process (step S210) shown in FIG. 4 different from FIG. 5 especially. It is. It is a flowchart showing operation | movement of the communication assistance system which concerns on 6th Embodiment of this invention, and is especially a flowchart for demonstrating the specific example of the stimulus amount calculation process (step S300) shown in FIG. It is a figure explaining the specific example of the avatar robot showing the 2nd participant shown in FIG. 1 used for the communication assistance system which concerns on 7th Embodiment of this invention. It is the figure shown with the graph for demonstrating concretely the casual range of distribution of the sense E of the 1st participant 13 of the 1st base 11A, and the 3rd participant. (A) is a graph for specifically explaining the casual range of the sensation E of the first base 11A from the distribution of the sensation E of the first participant 13 and the third participant 14 of the first base 11A. (B) explains how to calculate the sensation E that the second participant wants to give from the range where the distribution of the sensation E per participant is felt casual when there is a lot of participation at the first base 11A. It is the figure shown with the graph for doing. (A) is the figure shown with the graph for demonstrating the specific example of the probability density of the maximum value E_max of the sensation which the multiple participant who exists in the 1st base 11A feels casually, (b) is 1st. It is the figure explaining the specific example of the probability density of the minimum value E_min of the sensation that a plurality of participants existing in the base 11A feel casually, (c) according to the base situation information of the video conference of the first base 11A, It is the figure shown with the graph for demonstrating the specific example which updates the range E_max and E_min of the amount of sensations which the 1st base 11A feels casually. Reflecting the degree that the second participant 15 at the second base 11B wants to be aware of his / her presence, a specific example for calculating the sensory amount E to be presented to the first base participant It is the figure shown by the graph.

The present invention reduces the difference in presence due to the difference in remote space by presenting its presence casually to the other party without stopping the progress of conferences and conversations by communication between remote locations This makes it possible to provide a sense of unity of communication performed over a communication network.
In general, the amount of human sense has a relationship with the stimulus intensity. For example, according to Fechner's law, the sensory amount is proportional to the logarithm of the stimulus intensity.
Here, if the constant K and the stimulus amount R are given, the sensory amount (psychological amount) E can be expressed as E = K · log (R).
Therefore, if the amount of stimulation given to the remote participant is adjusted, it can be expected that the amount of presence can be adjusted, so that a casual presence can be presented to the remote participant. That is, it is possible to adjust the degree of presenting one's presence to the other party.

Embodiments of the present invention will be described with reference to the drawings.
<First Embodiment>
FIG. 1 is a schematic diagram for explaining an example of a configuration in which the communication assistance system according to the first embodiment of the present invention is used in a video conference system.
Here, with reference to FIG. 1, the functional configuration of the communication assist system will be described.
1 includes a video conference terminal (conference terminal) 10A and a video conference terminal (conference terminal) 10B installed at the first base 11A and the second base 11B, the video conference terminal 10A, and the video conference, respectively. The communication network N is connected to the terminal 10B.
The video conference terminal 10A and the video conference terminal 10B each include an imaging device 4, a microphone 5, a speaker 6, a display 7, and a processing device 8.

For example, an image (still image or moving image) captured by the imaging device 4 of the video conference terminal 10 </ b> A installed at the first base 11 </ b> A is subjected to a correction process on the image data by a cooperative operation between the imaging device 4 and the processing device 8. The image is displayed on the display 7 in the video conference terminal 10A. At the same time, the image data subjected to this correction processing is also transmitted to the video conference terminal 10B on the other side installed at the second base 11B connected to the communication network N, and an image is displayed on the display 7. .
For example, in the video conference terminal 10B installed at the second base 11B, when the operator presses an enlarge / reduce button of an operation unit (not shown), the same imaging device 4 provided in the video conference terminal 10B. Digital zoom processing is performed in cooperation with the processing device 8. As a result, the digital zoom image is displayed on the display 7 of the video conference terminal 10B. At the same time, the image data is also transmitted to the video conference terminal 10A installed at the first base 11A connected to the communication network N and displayed on the display 7 thereof.

In FIG. 1, a first participant 13 and a third participant 14 exist at the first base 11A, and a second participant 15 exists at the second base 11B. Further, an avatar robot 17 representing the second participant 15 existing at the second base 11B is movably disposed at the first base 11A.
The present invention is effective when three or more people communicate between two different sites. Hereinafter, a case where a video conference is performed using the video conference terminals 10 </ b> A and 10 </ b> B arranged at remote locations will be described as an example.
FIG. 1 is a specific example when communication is performed between different bases.
The situation where the conversation at the first base 11A was incandescent, that is, the situation where the first participant 13 and the third participant 14 were excited about the video conference conversation and forgot the existence of the second participant 15 and continued the conversation. Is assumed.
The second participant 15 speaks to the first participant 13 and the third participant 14, and interrupts during the conversation between the first participant 13 and the third participant 14, It is also possible to inform the third participant 14 of the presence of the second participant 15.
However, it is assumed that the second participant 15 does not want to stop the video conference, and does not want to speak. However, I want you to be aware of my existence. In such a case, the avatar robot 17 representing the second participant 15 present at the first base 11A casually presents the presence of the second participant 15, so that the progress of the video conference is not hindered. The presence of the two participants 15 can be communicated to the first participant 13 and the third participant 14. The present invention is used in the above situation.

In FIG. 1, the communication assistance system according to the first embodiment of the present invention is a communication assistance system such as a first site situation measurement estimation unit 20, a relay unit 30, a second site situation measurement estimation unit 40, and a stimulus amount calculation output unit 50. Equipment.
FIG. 2 is a block diagram showing the communication assist system according to the first embodiment of the present invention.
In FIG. 2, the communication assistance system includes a first site situation measurement estimation unit 20, a second site situation measurement estimation unit 40, and a stimulus amount calculation output unit 50.
The first site status measurement estimation unit 20 includes a first site status measurement unit 21, a first site status capture unit 23, a first site status estimation unit 25, and a wireless communication unit 27. The first site status measurement unit 21 measures the status of the first site 11A. The first site status capture unit 23 captures the status of the first site 11 </ b> A measured by the first site status measurement unit 21. The first site status estimating unit 25 estimates first site status information indicating the status information of the first site 11A from the status of the first site 11A captured by the first site status capturing unit 23. The wireless communication unit 27 transmits the first site status information estimated by the first site status estimation unit 25 to the radio communication unit 51. In this way, the first site status measurement estimation unit 20 estimates the first site status information representing the status of the first site 11A, and the radio communication unit 51 provided in the stimulus amount calculation output unit 50 from the radio communication unit 27. To send the first site status information.

The relay unit 30 wirelessly relays communication information transmitted / received via the communication network N between the second site situation measurement estimation unit 40 and the stimulus amount calculation output unit 50.
The second site status measurement estimation unit 40 includes a second site status measurement unit 41, a second site status capture unit 43, a second site status estimation unit 45, and a second site communication control unit 47. The second site status measurement unit 41 measures the status of the second site 11B. The second site status capturing unit 43 captures the status of the second site 11B measured by the second site status measuring unit 41. The second site status estimating unit 45 estimates second site status information indicating the status information of the second site 11B from the status of the second site 11B captured by the second site status capturing unit 43. The second site communication control unit 47 transmits the second site status information to the relay unit 30. As described above, the second site status measurement estimation unit 40 estimates the second site status information representing the status of the second site 11B, and transmits the second site status information from the second site communication control unit 47 to the relay unit 30. To do.
The relay unit 30 receives the second site status information transmitted from the second site communication control unit 47 via the communication network N, and transmits the second site status information to the stimulus amount calculation output unit 50.
The stimulus amount calculation output unit 50 includes a wireless communication unit 51, a wireless communication unit 53, a stimulus amount calculation unit 55, and a stimulus output unit 57.

The wireless communication unit 51 receives the first site status information from the first site status measurement estimation unit 20 and outputs the first site status information to the stimulus amount calculation unit 55.
The wireless communication unit 53 receives the second site status information from the second site status measurement estimation unit 40 and outputs the second site status information to the stimulus amount calculation unit 55.
The stimulus amount calculation unit 55 calculates a sensory amount range that can be perceived by the first participant at the first base 11A, second base state information, and the first participant's sensory amount range. The intention amount calculating unit 55b that calculates the amount of intention representing the degree of intention to participate in the conference that appears in the behavior of the second participant in the second base 11B, the stimulus that calculates the amount of stimulation according to the amount of intention An amount calculation unit 55c is provided.
The stimulus amount calculation unit 55 calculates the stimulus amount from the sense amount that the second participant 15 of the second base 11B wants to present to the participant at the first base 11A, but the present invention is not limited to this. It is not something. That is, based on the second site status information and the range of the amount of sensation of the first participant, the amount of intention representing the degree of intention to participate in the conference that appears in the behavior of the second participant at the second site 11B is calculated. It may be calculated by the unit 55b and output by the stimulus output unit 57 so that the first participant can perceive the stimulus according to the intention amount.
The stimulus output unit 57 outputs a stimulus corresponding to the stimulus amount so that the first participant can perceive it.
Each wireless communication unit provided in each unit is for performing data communication using wireless communication between units at the first base 11A. The wireless communication unit 27 provided in the first site situation measurement estimation unit 20 faces the wireless communication unit 51 provided in the stimulus amount calculation output unit 50. On the other hand, the relay unit 30 faces the wireless communication unit 53 provided in the stimulus amount calculation output unit 50.

Specifically, the first site situation measurement estimation unit 20 measures and estimates the statuses of the first participant 13 and the third participant 14 existing at the first site 11A shown in FIG. In addition, the first site status measurement estimation unit 20 indicates the status of the first participant 13, the third participant 14, and the avatar robot 17 (representing the second participant 15) existing in the first site 11A shown in FIG. Measure and estimate.
The first site situation measurement estimation unit 20 has a ROM, a RAM, and a CPU inside, reads the operating system OS from the ROM, expands it on the RAM, starts the OS, and reads the program from the ROM under OS management. The status data collection process is executed.
The second site status measurement estimation unit 40 measures and estimates the status of the third participant 14 existing at the second site 11B shown in FIG.
The second site status measurement estimation unit 40 has a ROM, a RAM, and a CPU inside, reads the operating system OS from the ROM, expands it on the RAM, starts the OS, and reads the program from the ROM under OS management. The status data collection process is executed.
The stimulus amount calculation output unit 50 is based on the situation of the first base 11A estimated by the first base situation measurement estimation unit 20 and the situation of the second base 11B estimated by the second base situation measurement estimation unit 40. The avatar robot 17 representing the second participant 15 shown in FIG. 1 calculates and outputs a stimulus amount for casually presenting the presence of the second participant 15.
The stimulus amount calculation output unit 50 has a ROM, a RAM, and a CPU inside, reads the operating system OS from the ROM, expands the RAM on the RAM, starts the OS, reads the program from the ROM under the OS management, and stimulates The amount calculation process is executed.

  The first site status measuring unit 21 and the second site status measuring unit 41 measure an audio signal and a video signal using, for example, a microphone and a camera. Alternatively, a voice signal and an acceleration signal are measured using a microphone and an acceleration sensor. The first site status measuring unit 21 and the second site status measuring unit 41 may use the same type of signal or different types of signals. Specifically, both the first site situation measuring unit 21 and the second site situation measuring unit 41 may measure an audio signal and a video signal using a microphone and a camera. Further, both the first site situation measuring unit 21 and the second site situation measuring unit 41 may measure a voice signal and an acceleration signal using a microphone and an acceleration sensor. Further, the first site situation measuring unit 21 may measure an audio signal and a video signal using a microphone and a camera, and the second site situation measuring unit 41 uses the microphone and an acceleration sensor to obtain an audio signal and an acceleration signal. You may measure. Further, the first site situation measuring unit 21 may measure an audio signal and an acceleration signal using a microphone and an acceleration sensor, and the second site situation measuring unit 41 uses the microphone and the camera to obtain an audio signal and a video signal. You may measure.

The first site status capturing unit 23 captures data obtained from the first site status measuring unit 21. The second site status capturing unit 43 captures data obtained from the second site status measuring unit 41.
Each of the first site status capturing unit 23 and the second site status capturing unit 43 removes noise in order to detect, for example, the speech content of the participant and the presence or absence of pronunciation of the audio signal. In addition, the signal is processed at an appropriate sampling period in order to detect the content of the participant's speech and the presence or absence of pronunciation. Also, pre-processing is performed to remove noise from the video signal and to easily capture data of posture and movement signals of the first participant 13, the second participant 15, the third participant 14, and the avatar robot 17 shown in FIG. Do. In addition, noise is removed from the acceleration signal, and preprocessing is performed to make it easier to extract the posture and movement signals of the first participant 13, the second participant 15, the third participant 14, and the avatar robot 17.

The first site status estimating unit 25 estimates the status of the first site 11A based on the signal obtained from the first site status capturing unit 23. The second site status estimating unit 45 estimates the status of the second site 11B based on the signal obtained from the second site status capturing unit 43.
For example, from the voice signal, the content of the speech of the first participant 13, the second participant 15 and the third participant 14 shown in FIG. 1, the time of speech, the time of speech, and the volume of speech The degree of incandescence of the video conference at the first site 11A, the status of participation in the video conference at the first site 11A, the status of participation in the video conference at the second site 11B, and the status of the second participant 15 are estimated.
Specifically, the first site situation estimation unit 25, for example, from the video signal, the body movement during the video conference of the first participant 13, the second participant 15, and the third participant 14 shown in FIG. Predetermined characteristic actions, specifically nodding, staring, cheek sticking, etc., identifying the speaker, identifying the point of the speaker's line of sight, and identifying the other party The situation of the video conference is estimated from the above.
For example, the situation of the first base 11A is estimated from the relative distance between the first participant 13 and the third participant 14 shown in FIG. 1 and the avatar robot 17 (robot unit) representing the second participant 15 from the video signal. To do. For example, from the acceleration signal, the body movement of the first participant 13, the second participant 15 and the third participant 14 during the video conference, a predetermined characteristic action, specifically, nodding, Estimate the status of video conferencing from actions such as staring and cheek sticks. Considering not only the presence / absence and number of times of motion and voice, but also the size of motion, it is reflected in the estimation of the situation.
Specifically, in the first site situation estimation unit 25, when the first participant 13 and the third participant 14 of the first site 11A perform an avatar robot 17 representing the second participant 15 or a remote video conference. The conversation can be continued only with the first participant 13 and the third participant 14 at the first base 11A without directing the line of sight to the projected image of the second participant 15 used in many cases. Then, the situation in which the speech toward the second participant 15 is not performed is estimated from the sound, the movement of the body, and the line of sight using the audio signal, the video signal, and the acceleration signal.

For example, in the first site situation estimation unit 25, from the length of time during which the conversation continues only with the first participant 13 and the third participant 14 of the first site 11A, the volume of the conversation, and the size of the operation, Forgetting the presence of the second participant 15 at the second base 11B, the degree of incandescence in the video conference is estimated. Further, the first site status estimation unit 25 calculates a stimulus amount calculation output described later from the distances between the first participant 13 and the third participant 14 at the first site 11A and the avatar robot 17 representing the second participant 15. The amount of stimulation output from the unit 50 is adjusted. For example, when the distance between the first participant 13 and the third participant 14 at the first base 11 </ b> A and the avatar robot 17 representing the second participant 15 is long, the avatar robot representing the second participant 15. Is estimated to be a situation in which the first participant 13 and the third participant 14 at the first base 11A are difficult to feel. When it is estimated that it is difficult to feel, the avatar robot 17 representing the second participant 15 outputs a large stimulus.
For example, in the 1st base situation estimation part 25, when the distance of the 1st participant 13 of the 1st base 11A, the 3rd participant 14, and the avatar robot 17 showing the 2nd participant 15 is near, it is 1st. It is estimated that the first participant 13 and the third participant 14 at the base 11A can easily feel. As a result, the avatar robot 17 representing the second participant 15 outputs a small stimulus. When it is estimated that it is easy to feel, this means that when the same stimulus is output, the avatar robot 17 representing the first participant 13, the third participant 14, and the second participant 15 at the first base 11 </ b> A. This is because the amount of sensation felt by the first participant 13 and the third participant 14 at the first base 11 </ b> A varies depending on the distance to the first base 11 </ b> A.

  In the second site status estimation unit 45, the second participant 15 of the second site 11B has not spoken for a predetermined time, and has a characteristic such as wearing a cheek stick, nodding or sighing. An action (behavior, behavior) is performed, and a situation in which the first participant 13 and the third participant 14 of the first base 11A want to be aware of their existence is expressed by an audio signal, a video signal, or an acceleration signal. Used to estimate from sound and body movements and line of sight. Considering not only the presence / absence and frequency of actions and voices but also the magnitude of the actions, it is reflected in the estimation of the base situation information. The degree to which the second participant 15 at the second base 11B wants to be aware of his / her presence from the length of time that the second participant 15 at the second base 11B is not speaking and the size of the movement. Is estimated.

The stimulus amount calculating unit 55 calculates the stimulus amount from the base state information estimated by the first base state capturing unit 23 and the base state information estimated by the second base state capturing unit 43. Specifically, the second participant 15 at the second base 11B is the base situation information that the first participant 13 and the third participant 14 at the first base 11A want to be aware of their existence. When it is estimated, it is the base situation information that the first participant 13 and the third participant 14 of the first base 11A are heated in the discussion and have forgotten the presence of the second participant 15 of the second base 11B. When estimated, in order to casually present the presence of the second participant 15 at the second base 11B to the first participant 13 and the third participant 14 at the first base 11A, the stimulation amount is calculated.
For example, the stimulus amount calculation unit 55 calculates the stimulus amount based on the base state information estimated by the first base state capture unit 23 and the base state information estimated by the second base state capture unit 43. The stimulus amount calculation unit 55 inclines the output stimulus amount by the first participant 13 and the third participant 14 at the first base 11A and forgets the presence of the second participant 15 at the second base 11B. The degree and the degree that the second participant 15 of the second base 11B wants the first participant 13 and the third participant 14 of the first base 11A to be aware of their presence are adjusted. For example, when the second participant 15 at the second base 11B wants to be aware of his / her presence, the stimulation amount is calculated from a predetermined stimulation range. When the second participant 15 at the second base 11B wants to be aware of his / her presence is small, a small stimulus amount is calculated from a predetermined stimulus range. When the first participant 13 and the third participant 14 at the first base 11A are incandescent in the discussion and forget about the presence of the second participant 15 at the second base 11B, Calculate a large amount of stimulation from the range. If the first participant 13 and the third participant 14 at the first base 11A are incandescent in the discussion and the degree of forgetting the presence of the second participant 15 at the second base 11B is small, Calculate a small amount of stimulus from the range.
For example, when the distance between the first participant 13 and the third participant 14 at the first base 11 </ b> A and the avatar robot 17 representing the second participant 15 is long, the avatar robot 17 representing the second participant 15. Calculates a large stimulus amount, and when the distance is short, the avatar robot 17 representing the second participant 15 calculates a small stimulus amount.

The stimulus amount calculation unit 55 is a unit that converts the calculated stimulus amount into a signal corresponding to the stimulus output unit 57. As will be described later, for example, when the stimulus output unit 57 uses vibration of a motor or the like, the amplitude and the frequency are calculated according to the calculated stimulus amount. For example, when the stimulus output unit 57 emits light such as a light emitting diode LED, the light amount, current, and voltage values to be output are calculated according to the calculated stimulus amount. When the stimulus output unit 57 outputs a color such as an LED, it outputs a shade of color to be output according to the calculated stimulus amount, such as dark red or light red, or a color change such as blue or red. The stimulus output unit 57 generates a light wave signal that can be sensed by visual observation by the first participant.
When the stimulus output unit 57 outputs sound from a speaker or the like, the sound output volume is calculated according to the calculated stimulus amount. When the stimulus output unit 57 outputs a sound from a speaker or the like, the language or voice content to be output or the volume thereof is calculated according to the calculated stimulus amount. Specifically, a language or voice such as “Ah” or “Nh” is selected, and the volume of the voice is calculated. The stimulus output unit 57 generates a sound wave signal that can be sensed by listening to the first participant.
The stimulus output unit 57 calculates the stimulus calculated by the stimulus amount calculation unit 55 based on the site status information estimated by the first site status capturing unit 23 and the site status information estimated by the second site status capturing unit 43. Output quantity. For example, the avatar robot 17 representing the second participant 15 shown in FIG. 1 is equipped with devices such as motors, LEDs, and speakers, and the above-mentioned devices are estimated by the first site status capturing unit 23. The stimulus amount calculated by the stimulus amount calculation unit 55 based on the information and the site state information estimated by the second site state capturing unit 43 is output.

FIG. 3 is a flowchart showing the operation of the communication assist system according to the first embodiment of the present invention.
First, in step S100, status input processing is performed. The first site status measurement unit 21, the first site status acquisition unit 23, the second site status measurement unit 41, and the second site status acquisition unit 43 shown in FIG. Measure and capture the status of the first base 11A and the second base 11B. Here, the input used for the situation input process is measurement data obtained from the first site situation measurement unit 21 and the second site situation measurement unit 41 shown in FIG.
In step S100, the obtained measurement data is converted into data used in step S200.
Specifically, for example, in step S100, the second participant 15 at the second base 11B wants the first participant 13 and the third participant 14 at the first base 11A to be aware of their presence. It is measured whether it is a situation and whether the first participant 13 and the third participant 14 at the first base 11A are in a heated discussion and have forgotten the presence of the second participant 15 at the second base 11B. This is the step to capture. For example, an audio signal and a video signal are input using a microphone and a camera. Alternatively, an audio signal and an acceleration signal are input using a microphone and an acceleration sensor. Here, noise is removed from the audio signal in order to detect the content of the participant's speech and the presence of pronunciation, or the signal is processed at an appropriate sampling period to detect the content of the participant's speech and the presence of pronunciation. Then, noise is removed from the video signal, and signals of posture and movement of the first participant 13, the second participant 15, the third participant 14, and the avatar robot 17 shown in FIG. 1 are extracted.
The output of step S100 is the data processed in step S100 for performing the situation input process.

In step S200 for performing the situation calculation process, the site status information of the first site 11A and the second site 11B shown in FIG. 1 is used by using the first site status estimation unit 25 and the second site status estimation unit 45 shown in FIG. Is estimated and calculated.
The input of step S200 is the output of step S100.
The process of step S200 estimates the site status information of the first site 11A and the second site 11B shown in FIG. 1 using the first site status estimation unit 25 and the second site status estimation unit 45 shown in FIG. calculate.
The output of step S200 is the estimated site status information of the first site 11A and the second site 11B shown in FIG.
Specifically, for example, in step S200, the second participant 15 at the second base 11B wants the first participant 13 and the third participant 14 at the first base 11A to be aware of their presence. The degree is calculated, and the degree to which the first participant 13 and the third participant 14 at the first base 11A are incite of discussion and forgetting the presence of the second participant 15 at the second base 11B is calculated.
The input in step S200 is appropriate for removing noise from audio signals, video signals, and acceleration signals obtained using, for example, a microphone, a camera, and an acceleration sensor, and detecting the participant's speech content and the presence or absence of pronunciation. It is a signal obtained by processing a signal at a sampling period.

In step S200, for example, the speech contents of the first participant 13, the second participant 15 and the third participant 14 shown in FIG. 1 from the audio signal, the time of speech, the time of speech, From the volume, the degree of incandescence of the video conference at the first site 11A shown in FIG. 1 and the site status information of participation in the video conference at the first site 11A shown in FIG. 1 are estimated. For example, from the video signal, the body movement during the video conference of the first participant 13, the second participant 15, and the third participant 14 shown in FIG. Estimate the video conference site status information from information such as nodding, staring, and cheek sticking, and identifying the point of gaze and who is talking to. For example, the base situation information is estimated in consideration of not only the presence / absence and number of times of the motion and voice, but also the size of the motion.
The output of step S200 is the estimated location status information of the video conference of the first location 11A shown in FIG. 1 and the location status information of the video conference of the first location 11A shown in FIG. For example, the degree of incandescence of the video conference at the first base 11A shown in FIG. 1 or the degree that the second participant 15 at the second base 11B wants to be aware of their presence. For example, whether the first participant 13 and the third participant 14 of the first base 11A shown in FIG. 1 are the base situation information estimated to have forgotten the presence of the second participant 15 of the second base 11B, Whether or not it is estimated that the participant at the second base 11B wants to be aware of his / her presence. For example, based on the distance between the avatar robot 17 representing the second participant 15 shown in FIG. 1 and the first participant 13 and the third participant 14 in the first site 11A, it is determined whether the site status information is easily noticed by the second participant 15. The value to represent.

In step S300 for performing the stimulus amount calculation process, the stimulus amount is calculated based on the site status information estimated by the first site status estimation unit 25 and the site status information estimated by the second site status estimation unit 45 shown in FIG. The stimulation amount is calculated using the unit 57.
The input of step S300 is the output of step S200. Step S300 uses the stimulus amount calculation unit 57 based on the site status information estimated by the first site status estimation unit 25 and the site status information estimated by the second site status estimation unit 45 shown in FIG. Calculate the amount of stimulation. The output of step S300 is a value representing the stimulus amount calculated in the above process.
Specifically, for example, in step S300, the amount of stimulation is calculated according to the video conference site status information of the first site 11A shown in FIG. 1 and the video conference site status information of the second site 11B shown in FIG. .
The input of step S300 is, for example, the base estimated that the first participant 13 and the third participant 14 of the first base 11A shown in FIG. 1 have forgotten the presence of the second participant 15 of the second base 11B. Whether it is situation information or whether it is estimated that the participant of the second base 11B wants to be aware of his / her existence. For example, based on the distance between the avatar robot 17 representing the second participant 15 shown in FIG. 1 and the first participant 13 and the third participant 14 in the first site 11A, it is determined whether the site status information is easily noticed by the second participant 15. The value to represent. For example, it is the site status information of the video conference of the first site 11A shown in FIG. 1 and the site status information of the video conference of the first site 11A of FIG. For example, the degree of incandescence of the video conference at the first base 11A shown in FIG. 1 or the degree that the second participant 15 at the second base 11B wants to be aware of their presence.
In step S300, based on the second site status information and the range of the amount of sensation of the first participant 13, a will amount representing the degree of intention to participate in the conference that appears in the behavior of the second participant 15 at the second site 11B. Is calculated.
In step S300, for example, the diagram shows the degree of incandescence of the video conference at the first base 11A shown in FIG. 1 or the degree that the second participant 15 at the second base 11B wants to be aware of their presence. The stimulation amount to be presented to the first participant 13 and the third participant 14 in the first base 11A shown in FIG.
The output of step S300 depends on, for example, the incandescence of the video conference at the first base 11A shown in FIG. 1 or the degree that the second participant 15 at the second base 11B wants to be aware of his / her presence. The amount of stimulation presented to the first participant 13 and the third participant 14 at the first base 11A shown in FIG.

In step S400 for performing the stimulus output process, a stimulus is output using the stimulus amount calculated by the stimulus amount calculation unit 55 shown in FIG.
The input of step S400 is the output of step S300. In step S400, the value output in step S300 is converted into the calculated input value of the stimulus output unit 57.
The output of step S400 is to actually output a stimulus using the stimulus amount calculated by the stimulus output unit 57 shown in FIG.
Specifically, for example, in step S400, the stimulus that is the output of step S300 is converted into a signal of a device that is actually used as the stimulus output, and the stimulus is actually output.
The input in step S400 depends on, for example, the incandescence of the video conference at the first base 11A shown in FIG. 1 or the degree that the second participant 15 at the second base 11B wants to be aware of his / her presence. The amount of stimulation presented to the first participant 13 and the third participant 14 at the first base 11A shown in FIG.

In step S400, for example, when the stimulus amount calculated by the stimulus output unit 57 shown in FIG. 2 is output to the vibration motor, the input stimulus amount is converted into an amplitude and a frequency. The amplitude may be fixed and the input stimulus amount may be converted into a frequency, or the frequency may be fixed and the input stimulus amount may be converted into an amplitude.
The output of step S400 is, for example, to vibrate the vibration motor with the amplitude and frequency converted in the above process when the stimulus amount calculated by the stimulus output unit 57 shown in FIG. 2 is output to the vibration motor. .
In addition, according to this embodiment, although the 2nd participant 15 of the 2nd base 11B calculates the stimulus amount from the sensory amount which he would like to show toward the participant in the 1st base 11A, this invention does this. It is not limited. That is, based on the second site status information and the range of the amount of sensation of the first participant, the amount of intention representing the degree of intention to participate in the conference that appears in the behavior of the second participant at the second site 11B is calculated. It may be calculated by the unit 55b and output by the stimulus output unit 57 so that the first participant can perceive the stimulus according to the intention amount. This adjusts the degree of presenting the amount of intention representing the degree of willingness to participate in the conference that appears in the speech of the remote participant based on the second site status information and the range of the sensory amount of the first participant. be able to.
In addition, based on the first site status information, the second site status information, and the range of the sense amount of the first participant, an intention amount representing the degree of intention to participate in the conference that appears in the speech and behavior of the remote participant is presented. The degree can be adjusted.

Second Embodiment
FIG. 4 is a flowchart showing the operation of the communication assistance system according to the second embodiment of the present invention, and in particular, is a flowchart showing a specific example of step S200 shown in FIG.
Step S210 is a subroutine for performing the first site status calculation process, and step S250 is a subroutine for performing the second site status calculation process. Step S200 shown in FIG. 3 is a step including a first site status calculation process S210 and a second site status calculation process S250.
The input in step S210 is information captured by the first site status capturing unit 23 illustrated in FIG. On the other hand, in step S210, the site status information of the first site 11A is calculated using the first site status estimation unit 25 shown in FIG. The output of step S210 is the site status information of the first site 11A calculated by the above processing.
Specifically, for example, the input in step S210 is information obtained by measuring the situation of the first participant 13 and the third participant 14 in the first base 11A shown in FIG. For example, it is data of an audio signal, a video signal, and an acceleration signal subjected to preprocessing such as noise removal and signal conversion into an appropriate sampling period. For example, the first participant 13, the second participant 15, and the third participant 14 shown in FIG. This is data such as the action of putting a cheek cane and the person who is talking to the person with the point of gaze specified.

In step S210, for example, the first participant 13 and the third participant 14 of the first base 11A are the avatar robot 17 representing the second participant 15, or the second participant 15 often present in a remote video conference. Continuing the conversation with only the first participant 13 and the third participant 14 at the first base 11A without giving a gaze to the image that projects the figure, and not making a statement toward the second participant 15 The site status information is estimated and calculated from sound, body movement, and line of sight using audio signals, video signals, and acceleration signals. For example, the distance between the first participant 13 and the third participant 14 at the first base 11A and the avatar robot 17 representing the second participant 15 is calculated.
The output of step S210 is a value representing the degree to which the first participant 13 and the third participant 14 at the first base 11A forget the presence of the second participant 15 at the second base 11B and are incandescent in the video conference. is there.
The input in step S250 is data captured by the second site status capturing unit 43 shown in FIG.
In step S250, the base state information of the second base 11B shown in FIG. 1 is estimated and calculated using the second base state estimation unit 45 shown in FIG.

The output of step S250 is data representing the base state information of the second base 11B shown in FIG.
Specifically, for example, the input in step S250 is information obtained by measuring the base state information of the second participant 15 in the second base 11B shown in FIG. For example, it is data of an audio signal, a video signal, and an acceleration signal subjected to preprocessing such as noise removal and signal conversion into an appropriate sampling period. For example, it is an audio signal of the second participant 15 at the second base 11B shown in FIG. 1, or a characteristic operation signal such as wearing a cheek stick, nodding or sighing.
In step S250, the base situation information of the second participant 15 in the second base 11B shown in FIG. 1 is calculated. For example, the participation of the second participant 15 of the second base 11B in the video conference does not speak for more than a predetermined time, such as wearing a cheek stick, nodding or sighing. The base situation information that the first participant 13 and the third participant 14 of the first base 11A want to be aware of the presence of the sound is transmitted using sound signals, video signals, and acceleration signals. This is a process of estimating and calculating from body movement and line of sight.
The output of step S250 is not until the second participant 15 of the second base 11B speaks, for example, but the first participant 13 and the third participant 14 of the first base 11A want to care about their existence. Or, it is a value that represents the degree of base situation information that you want to notice.
Thus, based on the second site status information and the range of the first participant's sense amount, the degree of presenting the amount of intention representing the degree of intention to participate in the conference that appears in the speech of the remote participant is adjusted. It can be carried out.

<Third Embodiment>
FIG. 5 is a flowchart showing the operation of the communication assistance system according to the third embodiment of the present invention, and in particular, is a flowchart showing a specific example of the first site situation calculation process (S210) shown in FIG.
First, the flowchart shown in FIG. 5 will be schematically described.
In step S211, it is determined whether the second participant 15 has spoken for a predetermined time. If the second participant 15 has not spoken for a predetermined time or longer, the process proceeds to step S212, and variables are initialized. In step S213, it is determined whether there is a characteristic action in the behavior of the participant. If there is a characteristic action, the process proceeds to step S214, where the speaker is identified and the line of sight is detected.
In step S <b> 215, it is determined whether the point of the speaker's line of sight is other than the place corresponding to the second participant 15. If the point of sight of the speaker is other than the place corresponding to the second participant 15, the process proceeds to step S216, and the number of detected characteristic actions of the first base 11A is updated. Next, in step S217, the detected total amount of characteristic operations of the first base 11A is updated. Next, in step S218, the loop variable is updated. Next, in step S219, it is determined whether the time taken into consideration in the next step is smaller than zero.
When the time to be considered in the next step is smaller than 0, the process proceeds to step S220, and it is determined whether the number of detected characteristic actions of the first base 11A is larger than a predetermined threshold value of the number of characteristic actions.
When the number of characteristic actions of the first site 11A is larger than a predetermined threshold value of the number of characteristic actions, the process proceeds to step S221, and a flag representing the site status information of the first site 11A is calculated. Next, in step S222, the calculated data is output.

Here, the definition of variables used in the flowchart shown in FIG. 5 will be described.
Time_current represents the current time. Time_interval is a value that represents a predetermined time interval, and is used to detect a characteristic action at each time interval from the current time. Time_nonVoice is a value that represents a predetermined length of time, and is used in the step of determining whether the second participant 15 is in a silent section greater than this value. Flag_11A is a flag representing the base state information of the first base 11A, and takes a value of 0 or 1.
At 1, the site status information of the first site 11A represents a state in which the site status information of the second site 11B is forgotten and the discussion is heated. Otherwise, it takes a value of 0. j represents a variable of the loop. MotionValue_average represents an average value of predetermined characteristic motion magnitudes. MotionCTS_Threshold represents a predetermined threshold value of the number of characteristic operations. MotionValue_detection represents the amount of characteristic motion detected. MotionValue_11A represents the total amount of characteristic operations of the detected first base 11A. MotionCTS_11A represents the number of detected characteristic operations of the first base 11A.

In the flowchart shown in FIG. 5, the input in step S211 for determining whether or not the second participant 15 has spoken for a predetermined time or more is data obtained by measuring the site status information of the second site 11B. For example, it is an audio signal measured by the microphone of the second participant 15 at the second base 11B.
The process of step S211 for determining whether or not the second participant 15 has spoken for a predetermined time or more is a process for determining whether or not the second participant 15 has spoken for a predetermined time Time_nonVoice or more. When the second participant 15 does not utter a sound higher than a predetermined volume from the current time Time_current to Time_current-Time_nonVoice, it is determined that the second participant 15 does not speak for a predetermined time or more. . The output of step S211 for determining whether or not the second participant 15 has spoken for a predetermined time is “Yes” or “No”. If “yes”, the process proceeds to S212, and if “no”, the process proceeds to an end step. In addition, the current time Time_current is output.
Next, the input of step S212 for initializing variables is the output of S211. The process of step S212 for initializing variables is a process for initializing variables. Specifically, a flag Flag_11A representing the base state information of the first base 11A, a detected total amount of characteristic motion of the first base 11A, MotionValue_11A, and a detected number of characteristic actions of the first base 11A, MotionCTS_11A, The value of the variable j of the loop is set to 0. The output of step S212 for initializing variables is Flag_11A, MotionValue_11A, MotionCTS_11A and j, and the value of Time_current output in S211.

Next, the input of step S213 for determining whether or not there is a characteristic action is the output of S212 and a signal of a predetermined normalized characteristic action. The process of step S213 for determining whether or not there is a characteristic action is predetermined between time {Time_current−j × predetermined time interval Time_interval} and time {Time_current− (j + 1) × Time_interval}. This is a process for determining whether or not there is a characteristic action.
Specifically, when using the signal of the acceleration sensor, a predetermined normalized characteristic operation, for example, an operation of nodding is prepared, and signals of x, y, and z of the acceleration sensor are prepared, and pattern recognition processing is performed. Thus, it is determined whether or not there is a signal that matches the nodding operation signal in the time interval. The predetermined characteristic operation is not limited to one operation, and a plurality of operations may be prepared. It is necessary to prepare as many actions as you want to detect.

In this flowchart, the case where there is one characteristic operation to be detected is described as an example. The output of step S213 for determining whether or not there is a characteristic action is “Yes” or “No”. If a characteristic action is extracted by the above process, “Yes” is output, and if it is not, “No” is output. If “yes”, the process proceeds to step S214. If “no”, the process proceeds to step S218.
In addition, the type, number of times, and the size of characteristic action information that is characteristic action data obtained by the above processing are output. Specifically, the magnitude of the characteristic action is held in a variable of the detected characteristic action amount MotionValue_detection. For example, when nodding loudly, the signal is larger than a signal of a predetermined normalized acceleration sensor. The magnitude is calculated in the process of step S213 based on the acceleration sensor signal, specifically the data such as the amplitude of the waveform, and output.
In addition, the values of j, Flag_11A, MotionValue_11A, MotionCTS_11A, and Time_current are also output. The values of these variables may be output each time or may be managed in a database. In the future, output and input data will be omitted as necessary.

Next, the input in step S214 for performing the speaker identification and line-of-sight detection processing is the output in step S213. The process of step S214 for performing the process of specifying the speaker and detecting the line of sight is a process of specifying the speaker who is speaking at the first base 11A and specifying the speaking partner of the speaker who is speaking by detecting the line of sight. .
Specifically, for example, by voice signal, which one or both of the first participant 13 and the third participant 14 at the first base 11A are talking, or no one is talking. Determine. When one or more utterances are confirmed, a line of sight is detected from the video signal, and a process of specifying the speaking partner is performed. The output of step S214, which performs speaker identification and line-of-sight detection processing, is data representing the speaker and the point of the line of sight.
The input of step S215 for determining whether the point of the speaker's line of sight is outside the place corresponding to the second participant 15 is the output of step S214.

The process of step S215 for determining whether the point of the speaker's line of sight is other than the place corresponding to the second participant 15 is the output of step S214. The point of the speaker's line of sight represents the second participant 15. This is a process for determining whether or not a thing is present. The output of step S215 for determining whether the point of sight of the speaker is outside the place corresponding to the second participant 15 is “Yes” or “No”. If “yes”, the process proceeds to S220, and if “no”, the process proceeds to an end step.
For example, it is assumed that an avatar robot 17 representing the second participant 15 exists in the first base 11A shown in FIG. Specifically, when the speaker's line of sight is the avatar robot 17, the determination is “No” because the speaker's line of sight represents the second participant 15. For example, as used in many remote video conferences, when a remote participant is shown on the screen, if the speaker's line of sight is the screen, the determination is “No”. Conversely, when the speaker is speaking to a participant who is at the same base as himself, specifically, the first participant 13 at the first base 11A speaks to the third participant 14. If yes, the determination is “yes”.

Next, the input of step S216 for updating the number of detected characteristic operation information of the first base 11A is the output of step S215, “Yes”. In addition, the characteristic operation information data detected in step S213. The process of step S216 for updating the number of detected characteristic motion information of the first base 11A is a process of updating the value of the detected number of characteristic motion information of the first base 11A MotionCTS_11A.
Specifically, for example, when there is only one characteristic motion information detected in step S213, 1 is added to the value of MotionCTS_11A. If there is one or more characteristic motion information detected in step S213, a value may be added according to the number. The output of step S216 for updating the number of detected characteristic operation information of the first base 11A is the updated value of MotionCTS_11A.
Next, the input in step S217 for updating the total amount of characteristic operation information of the first base 11A detected is the data of characteristic operation information detected in step S213. Specifically, for example, the amount of detected characteristic motion information Motion_detection. The process of step S217 for updating the detected total amount of characteristic operation information of the first base 11A performs a process of updating the total amount of characteristic operation information MotionValue_11A from the input data in step S217.

For example, when the number of characteristic motion information detected in step S213 is one, specifically, when the average value of the size of the detected characteristic motion information is MotionValue_average, the first detected The total value MotionValue_11A of the characteristic operation information of the base 11A is obtained by adding the value of MotionValue_detection / MotionValue_average to the current value of MotionValue_11A.
When the characteristic motion information detected in step S213 is large, a large value is added to the value of MotionValue_11A. This is a value that reflects the fact that the body movement increases as the discussion gets heated. The output of step S217 for updating the detected total amount of characteristic operation information of the first base 11A is the total amount MotionValue_11A of the characteristic operation information of the first base 11A calculated and updated by the above-described processing.

Next, the input of step S218 for updating the loop variable is the current value of j. The process of step S218 for updating the loop variable is a process of adding 1 to j. The output of step S218 for updating the loop variable is the updated value of j. This is a process performed to update the time interval for detecting characteristic motion information in step S213.
The input of step S219 for determining whether the time taken into consideration in the next step is smaller than 0 is j, Time_current and Time_interval which are the outputs of step S218. The process of step S219 for determining whether the time to be considered in the next step is smaller than 0 is a step for determining whether the time interval for detecting characteristic motion information in step S213 performed in the next step is smaller than 0. is there. Specifically, it is determined whether the value of Time_current− (j + 1) × Time_interval is smaller than zero. If it is less than 0, “Yes” is set, and if it is 0 or more, “No” is set.
The output of step S219 for determining whether the time to be considered in the next step is less than 0 is “Yes” or “No”. If “yes”, the process proceeds to step S220, and if “no”, the process proceeds to step S213.
Next, the input in step S220 for determining whether or not the number of detected characteristic operation information of the first base 11A is greater than a predetermined threshold value of the number of characteristic operation information is “Yes” in step S219, and MotionCTS_100. This is the value of the threshold value MotionCTS_Threshold of the number of times of predetermined characteristic motion information.

In the process of step S220 for determining whether or not the number of detected characteristic motion information of the first base 11A is larger than a predetermined threshold value of the number of characteristic motion information, the value of MotionCTS_100 updated in step S216 is MotionCTS_Threshold. A process for determining whether the value is larger is performed.
When the value of MotionCTS_100 is larger than MotionCTS_Threshold, it is determined as “Yes”, and when it is smaller, it is determined as “No”. For example, a large number of characteristic motion information was detected in a state where the first participant 13 at the first base 11A turned his gaze to something other than the one representing the second participant 15 in steps S214 and S215. In this case, specifically, the second participant 15 at the second base 11B is forgotten and there is a possibility that the discussion is incandescent. When the number of times is equal to or greater than a predetermined number, processing is performed to determine that the discussion is in a heated state. The output of step S220 for determining whether the number of detected characteristic operation information of the first base 11A is greater than a predetermined threshold value of the number of characteristic operation information is “Yes” determined in the above process, or “No”.

The input of step S221 for calculating the flag representing the site status information of the first site 11A is the output of step S220, “Yes”. The process of step S221 for calculating the flag indicating the site status information of the first site 11A is a process of calculating the value of the flag Flag_11A indicating the site status information of the first site 11A. Flag_11A takes a value of 0 or 1. When Flag_11A is 1, the base state information of the first base 11A represents a state in which the situation of the second base 11B is forgotten and heated to discussion. Otherwise, it takes a value of 0. Specifically, for example, when it is determined in step S220 that the second participant 15 at the second base 11B is forgotten and the discussion is heated, a value of 1 is set. In other cases, a process of setting a value of 0 is performed. The output of step S221 for calculating the flag representing the site status information of the first site 11A is the value of Flag_11A.
The input of step S222 for outputting the calculated data is the values of Flag_11A, MotionCTS_11A, and MotionValue_11A. The process of step S222 for outputting the calculated data is to output the input data. A process for outputting the data value calculated and updated in the previous steps is performed. The output of step S222 for outputting the calculated data is the values of Flag_11A, MotionCTS_11A, and MotionValue_11A.

For example, when the status of the first site 11A is in a state where the status of the second site 11B is forgotten and the discussion is heated, specifically, Flag_11A takes a value of 1. When the degree of incandescence is large, the values of MotionCTS_11A and MotionValue_11A are large, and when the degree of incandescence is small, the values of MotionCTS_11A and MotionValue_11A are small.
In this way, the first characteristic action information, the speaker information, and the conversation partner information are used as the first site situation information, and the will amount representing the degree of intention to participate in the conference that appears in the behavior of the remote participant The degree of presenting can be adjusted.
In addition, when the audio signal of the second base 11B within a predetermined time is below a predetermined volume, the second characteristic motion information is extracted based on the video signal of the second base 11B. It is possible to calculate the amount of intention that represents the degree of intention to participate in the conference that appears.

<Fourth embodiment>
FIG. 6 is a flowchart showing the operation of the communication assistance system according to the fourth embodiment of the present invention, and in particular, for explaining a specific example of the second site situation calculation process (step S250) shown in FIG. It is a flowchart.
First, the flowchart shown in FIG. 6 will be schematically described.
In step S251, it is determined whether the second participant has spoken for a predetermined time. In step S252, variables are initialized. In step S253, it is determined whether there is characteristic operation information. In step S254, the number of detected characteristic operation information of the second base 11B is updated. In step S255, the total amount of characteristic operation information of the detected second base 11B is updated. In step S256, the loop variable is updated. In step S257, it is determined whether the time taken into consideration in the next step is less than zero. In step S258, it is determined whether the number of detected characteristic motion information of the second base 11B is greater than a predetermined threshold value of the number of characteristic motion information. In step S259, a flag representing the base state information of the second base 11B is calculated. In step S260, the calculated data is output.

Here, the definition of variables used in the flowchart shown in FIG. 6 will be described.
Of the variable definitions, the variables described with reference to FIG. 5 are omitted.
Flag_11B is a flag representing the base state information of the second base 11B. It takes a value of 0 or 1. At 1, the site status information of the second site 11B represents the site status information that the user wants to be aware of, but not to speak to the participants at the first site 11A. Otherwise, it takes a value of 0. i represents a variable of the loop. MotionValue_11B represents the total amount of characteristic operation information of the detected second base 11B. MotionCTS_11B represents the number of detected characteristic operation information of the second base 11B.
Step S251 for determining whether or not the second participant 15 has spoken for a predetermined time or more is the same as step S211 shown in FIG.
Step S252 for initializing variables has the same contents as step S212 shown in FIG. In step S252, the total amount of detected characteristic motion information of the second base 11B, MotionValue_11B, the number of detected characteristic motion information of the second base 11B, MotionCTS_11B, the loop variable i, and the base state of the second base 11B The flag Flag_11B representing the information is used. The MotionValue_11B corresponds to the MotionValue_11A in step S212, the MotionCTS_11B corresponds to the MotionCTS_11A in step S212, i corresponds to j in the step S212, and Flag_11B corresponds to Flag_11A in the step S212.

Step S253 for determining whether or not there is characteristic operation information is the same as step S213 shown in FIG. However, in step S213 shown in FIG. 4, characteristic operation information representing the state in which the first participant 13 and the third participant 14 of the first base 11A are incandescent in the video conference is extracted. The characteristic operation information targeted in S253 is characteristic operation information that is displayed when the second participant 15 at the second base 11B wants to know his / her presence. For example, it is characteristic operation information that is shown when the user thinks he / she is in a video conference, such as a sighing operation.
Next, Step S254 for updating the number of detected characteristic operation information of the second base 11B is the same as Step S216 in FIG. However, the variable to be used is a variable related to the second base 11B described in the definition of the variable in FIG.
Step S255 for updating the detected total amount of characteristic operation information of the second base 11B is the same as step S217 shown in FIG. However, the variable to be used is a variable related to the second base 11B described in the definition of the variable in FIG.
Step S256 for updating the loop variable has the same contents as step S218 shown in FIG. However, the variable used is i.
Step S257 for determining whether the time to be considered in the next step is less than 0 is the same as step S219 shown in FIG.

Next, step S258 for determining whether or not the number of detected characteristic motion information of the second base 11B is greater than a predetermined threshold value of the number of characteristic motion information is the same as step S220 shown in FIG. . However, the variable to be used is a variable related to the second base 11B described in the definition of the variable in FIG.
Next, Step S259 for calculating a flag representing the base state information of the second base 11B has the same contents as Step S221 in FIG. However, the variable to be used is a variable related to the second base 11B described in the definition of the variable in FIG. Flag_11B is a flag representing the base state information of the second base 11B. At 1, the site status information of the second site 11B represents the site status information that the user wants to be aware of, but not to speak to the participants at the first site 11A. Otherwise, it takes a value of 0.
Next, Step S260 for outputting the calculated data has the same contents as Step S222 shown in FIG. However, the variable to be used is a variable related to the second base 11B described in the definition of the variable in FIG. Specifically, for example, the second participant 15 at the second base 11B does not speak to the first participant 13 and the third participant 14 at the first base 11A, but wants to present his / her presence. When the degree of thinking is large, the values of MotionCTS_11A and MotionValue_11A are large and not enough to speak, but when the degree of wanting to present their presence is small, the values of MotionCTS_11A and MotionValue_11A The value becomes smaller.
As described above, when the audio signal of the second base 11B within a predetermined time is below a predetermined volume, the second characteristic operation information is extracted based on the video signal of the second base 11B. It is possible to calculate the amount of intention representing the degree of intention to participate in the conference that appears in

<Fifth Embodiment>
FIG. 7 is a flowchart showing the operation of the communication assistance system according to the fifth embodiment of the present invention. In particular, a specific example of the first site situation calculation process (step S210) shown in FIG. 4 different from FIG. It is the flowchart demonstrated.
In step S223, a process of calculating a distance Distance_17 between the participant at the first base 11A and the avatar robot 17 is performed. In step S224, Flag_11A, MotionCTS_11A, MotionValue_11A, and Distance_17 are output.
Here, the definition of variables used in the flowchart shown in FIG. 7 will be described.
Note that variables already defined in FIGS. 5 and 6 are omitted.
Distance_17 represents the distance between the participant at the first site 11A and the avatar robot 17.
The flowchart shown in FIG. 7 is almost the same as the flowchart shown in FIG. The difference is that there is step S223 for performing a process of calculating the distance Distance_17 between the participant at the first base 11A and the avatar robot 17, which did not exist in the flowchart shown in FIG. Further, instead of step S222 shown in FIG. 5, step S224 in which the distance Distance_17 between the participant at the first base 11A and the avatar robot 17 is added to the data outputted in step S222 shown in FIG. That is.

In the step of measuring and calculating the situation of the first base 11A, for example, in addition to the situation of step S213, the first participant 13 and the third participant 14, the situation of the avatar robot 17 representing the second participant 15 is also measured. However, by calculating the site status information in a later step, the site status information of the first site 11A can be calculated in more detail than in the case shown in FIG.
Specifically, in step S213, the distance between the first participant 13 at the first base 11A and the avatar robot 17 representing the second participant 15 and the third participant 14 at the first base 11A are determined from the video data. The distance to the avatar robot 17 representing the second participant 15 is measured, the average of the measured distances is calculated, and the process of substituting for the variable Distance_17 is performed.
For example, not only the average of distances but also the closest distance between the avatar robot 17 representing the second participant 15 and the participants at the first base 11A may be substituted into the variable Distance_17. The far distance may be substituted into the variable Distance_17, or a value calculated using a predetermined function for the distance may be substituted.

The distance Distance_17 between the participant in the first site 11A and the avatar robot 17 is equal to the distance 1 from the first site 11A, even if the avatar robot 17 that outputs the stimulus outputs the same degree of stimulus. If the distance between the participant who is in the room and the avatar robot 17 is long, the amount of sensation felt by the participant at the first base 11A becomes small.
Conversely, if the value of Distance_17 is small, that is, if the distance between the participant at the first base 11A and the avatar robot 17 is short, the amount of sensation felt by the participant at the first base 11A increases. It is a value used to This is a step necessary to enable adjustment of the amount of stimulation to be presented to the participant at the first base 11A according to the distance Distance_17 between the participant at the first base 11A and the avatar robot 17. The above example relates to distance, but values other than distance may be used. For example, when the avatar robot 17 representing the participant of the first base 11A and the second participant 1002 is moving together, the above steps may be performed with a relative speed.
In this way, by using the distance between the first participant and the robot unit 17 as the first site status information, an intention amount representing the degree of willingness to participate in the conference that appears in the behavior of the remote participant is presented. The degree can be adjusted.

<Sixth Embodiment>
FIG. 8 is a flowchart showing the operation of the communication assistance system according to the sixth embodiment of the present invention, and in particular, a flowchart for explaining a specific example of step S300 for performing the stimulus amount calculation process shown in FIG. is there.
First, the flowchart shown in FIG. 8 will be schematically described.
In step S301, a flag is determined. In step S302, data is captured. In step S303, a process of calculating the range of the sensory amount from the site status information of the first site 11A is performed. In step S304, a process for calculating the sensory amount E from the site status information of the second site 11B is performed. In step S305, the stimulus amount R is calculated.
In step S306, signal conversion processing for the stimulus output unit 57 is performed. In step S307, output processing to the stimulus output unit 57 is performed.

Here, the definition of variables used in the flowchart shown in FIG. 8 will be described.
In addition, with reference to FIG.5, FIG.6, FIG.7, it omits about the predefined variable.
E represents the sensory amount of the first participant 13. E_max represents the maximum value of the sensory threshold. E_min represents the minimum value of the sensory threshold. E_max_basis represents the maximum threshold value of the sensory amount that is a predetermined reference. E_min_basis represents the minimum threshold value of the sensory amount that is a predetermined reference. MotionValue_11A_basis represents a value representing the base state information of the first base 11A that is a predetermined reference. MotionCTS_11A_basis represents a value representing the base state information of the first base 11A that is a predetermined reference. Motion_max_basis represents the maximum value of the base state information of the second base 11B, which is a predetermined reference. Motion_min_basis represents the minimum value of the base state information of the second base 11B, which is a predetermined reference. W_cts represents a weighting value. W_value represents a weighting value. R represents the amount of stimulation. K represents a predetermined constant. R_basis represents the reference value of the stimulation amount that has been determined in advance.
A represents an amplitude value. A_basis represents a reference value of a predetermined amplitude.

The input of the step S301 for determining the flag is the flag Flag_11A indicating the base state information of the first base 11A that is the output of the step S222 shown in FIG. 5, and the base of the second base 11B that is the output of the step S260 shown in FIG. It is a flag Flag_11B that represents status information. The process of step S301 for determining the flag is a step of determining whether the multiplication of Flag_11A and Flag_11B is 1. When the multiplication is 1, when the first participant 13 and the third participant 14 at the first base 11A are incandescent in the video conference, and the second participant 15 at the second base 11B is himself It is a situation that I want you to become aware of. The output of step S301 for determining the flag is “Yes” or “No”. In the above processing, “Yes” is output when the product of the multiplication is 1, and “No” is output otherwise. If “yes”, the process proceeds to step S302, and if “no”, the process proceeds to an end step.
The input of step S302 for capturing data is the output of step S222 shown in FIG. 5 and the output of step S260 shown in FIG. If the distance_17 is also measured, the output of step S224 shown in FIG. 7 is used instead of step S260 shown in FIG. The process of step S302 for fetching data is a process for fetching data. More specifically, these variables include MotionCTS_11A, MotionCTS_11B, MotionValue_11A, MotionValue_11B, and, in some cases, Distance_17. The output of step S302 for fetching data is the output of the variable. For example, data representing the site status information of the first site 11A, specifically, data representing the degree to which the first participant 13 and the third participant 14 of the first site 11A are incandescent in the video conference, This is a step of taking in data representing the degree to which the second participant 15 at the two bases 11B wants to know his / her presence.

Next, the input of step S303 for performing the process of calculating the range of the sensory amount from the site status information of the first site 11A is data representing the site status information of the first site 11A, for example, the first participant 13 of the first site 11A. And the values MotionValue_11A, MotionCTS_11A, and the maximum value E_max of the sensory amount threshold and the minimum value E_min of the threshold of the sensory amount.
In addition, the data MotionValue_11A_basis, the MotionCTS_11A_basis representing the degree to which the participants at the first base 11A serving as the reference are incandescent in the video conference, the threshold value of the reference sensory amount, E_max_basis, E_min_basis, in advance, W_cts and W_value are also prepared as input data as weighting values.
The process of step S303 that performs the process of calculating the range of the sensory amount from the situation of the first base 11A represents the degree to which the first participant 13 and the third participant 14 of the first base 11A are incandescent in the video conference. This is a step of performing processing for calculating and updating the maximum value E_max of the threshold value of the sensory amount and the minimum value E_min of the threshold value of the sensory amount according to the data. For example, the following values (1) and (2) are used to calculate and update the maximum value E_max of the sensory amount threshold value and the minimum value E_min of the sensory amount threshold value.

E_max = {W_cts × (MotionCTS_11A ÷ MotionCTS_11A_basis) + W_value (MotionValue_11A ÷ MotionValue_11A_basis)} × E_max_basis (1)
E_min = {W_cts * (MotionCTS_11A / MotionCTS_11A_basis) + W_value (MotionValue_11A / MotionValue_11A_basis)} * E_min_basis (2)

It is assumed that W_cts and W_value take a value that becomes 1 and take a value from 0 to 1. If it does so, the ratio of MotionCTS_11A and MotionValue_11A reflected on the threshold value of sensory amount can be adjusted. The threshold values of the reference sensory amount, E_max_basis and E_min_basis, satisfy Emax_basis> E_min_basis. Also, E_max_basis and E_min_basis must not be too large and too small.
Specifically, for example, a value of a sensory amount that a human feels casually under a predetermined condition, for example, a state in which something that vibrates with a small amplitude is in a position that enters the human visual field For example, E_max_basis represents the amount of sensation given by vibration of amplitude that can sense that 8 out of 10 people vibrate, and the amount of sensation given by vibration of amplitude that can sense that 2 out of 10 people vibrate. E_min_basis.
E_max_basis and E_min_basis must not be a sensory amount that cannot be sensed by humans, and cannot be a sensory amount by which anyone can reliably sense a large amount. When it vibrates with a large amplitude that everyone notices, it is not in a casual state.
The same applies to the values of E_max and E_min calculated based on the values of E_max_basis and E_min_basis. The above formula is an example in which the values of MotionCTS_11A and MotionValue_11A are weighted and the added value is reflected in the threshold value of the sensory amount.
The threshold value of the sensory amount may be calculated using another relational expression. The output of step S303 for performing the process of calculating the range of the sensory amount from the site status information of the first site 11A is calculated by the above process, and the updated maximum value E_max of the sensory amount threshold and the minimum of the sensory amount threshold are calculated. This is the value E_min. Specifically, if data indicating the site status information of the first base 11A, for example, the first participant 13 and the third participant 14 of the first base 11A are incandescent in the video conference, E_max and E_min Both values tend to increase. Specifically, if the degree of the data indicating the base state information of the first base 11A, for example, the first participant 13 and the third participant 14 of the first base 11A is incandescent in the video conference is low, E_max and E_min The value of is set small.

  Next, the input of step S304 for performing the process of calculating the sensation amount E from the site status information of the second site 11B is specifically, for example, the values of E_max and E_min that are the outputs of step S302, and predetermined values. The maximum value Motion_max_basis of the base status information of the second base 11B serving as a reference, the minimum value Motion_min_basis of the base status information of the second base 11B serving as a predetermined reference, and MotionCTS_11B and MotionValue_11B representing the base status information of the second base 11B And weight values W_cts and W_value. The process of step S304, which performs the process of calculating the sensory amount E from the site status information of the second site 11B, is a step of calculating the sensory amount E to be presented to the participant at the first site 11A from the input data. For example, it is a process for obtaining the sensory amount E by the following equation.

E = {(W_value × MotionValue — 11B + W_cts × MotionCTS — 11B) ÷ (Motion_max_basis + Motion_min_basis)} × (E_max + E_min) (3)
In the part of (W_value × MotionValue_11B + W_cts × MotionCTS_11B) shown in the above equation (3), the number of times and the number of times the degree of desire that the second participant 15 wants to be aware of the presence of the second base 11B The amount is weighted by the weighting number W, and the base state information of the second base 11B is digitized.

In the part of ÷ (Motion_max_basis + Motion_min_basis)} shown in the above equation (3), the maximum value Motion_max_basis of the base state information of the second base 11B serving as the predetermined reference and the base state of the second base 11B serving as the predetermined reference Compared with the value of the information minimum value Motion_min_basis, the degree of the value obtained by quantifying the base state information of the second base 11B is calculated.
In the part of x (E_max + E_min) shown in the above equation (3), it is calculated how much the numerical value of the base state information of the second base 11B is within the range of the sensory amount updated in step S303. The degree of the calculated value is calculated.
However, it is necessary to satisfy Motion_max_basis> Motion_min_basis. In short, it is a step of performing a process of calculating how much the base state information of the second base 11B is calculated from the range of the predetermined Motion_basis and converting the degree into the range of the sensory amount. The above formula is merely an example, and other relational expressions may be used. The output of step S304 for performing the process of calculating the sensory amount E from the base state information of the second base 11B is the value of the sensory amount E.

The input in step S305 for calculating the stimulus amount R is, for example, the value of the sensory amount E calculated in step S304 and the value of a predetermined constant K. The process of step S305 for calculating the stimulus amount R is, for example, using the Feffiner's law that defines the relationship between the human sense amount and the given stimulus amount, and the stimulus amount R = 10 ^ (sensory amount E / predetermined The sensory amount R is calculated based on the constant K).
The above equation (3) is an equation for calculating the stimulation amount R from the sensory amount E that the second participant 15 of the second base 11B wants to present to the participant at the first base 11A. If the degree of desire that the participant at the second base 11B wants to present his / her presence is large, the value of the sensory amount E is greatly calculated in step S304, and therefore the stimulus amount R calculated in step S305 is also large. . In addition, when the second conference 15 is forgotten at the first base 11A and the video conference is incandescent, the sensory amount ranges E_max and E_min calculated in step S303 are calculated to be large, so in step S304. The sensory amount E calculated is also increased, and the stimulus amount R calculated in step S305 is also increased. The reverse is true when the stimulus amount R is small. In the above description, the stimulus amount R is calculated from the sensory amount E using Feffiner's law, but other conversion formulas may be used. The output of step S305 for calculating the stimulus amount R is the value of the stimulus amount R.
According to the present embodiment, the stimulation amount R is calculated from the sensory amount E that the second participant 15 at the second base 11B wants to present to the participant at the first base 11A. It is not limited to this. That is, based on the second site status information and the range of the amount of sensation of the first participant, the amount of intention representing the degree of intention to participate in the conference that appears in the behavior of the second participant at the second site 11B is calculated. It may be calculated by the unit 55b and output by the stimulus output unit 57 so that the first participant can perceive the stimulus according to the intention amount. This adjusts the degree of presenting the amount of intention representing the degree of willingness to participate in the conference that appears in the speech of the remote participant based on the second site status information and the range of the sensory amount of the first participant. be able to.

Next, the input of step S306 that performs signal conversion processing for the stimulus output unit 57 is the value of the stimulus amount R that is the output of step S305. The process of step S306 for performing the signal conversion process for the stimulus output unit 57 is a process of converting the stimulus amount R into an amplitude and a frequency when the stimulus output unit 57 is a vibration motor, for example. For example, it is a process of converting the stimulus amount R into an amplitude. For example, it is a process of converting the stimulus amount R into a frequency. For example, when there is a reference amplitude A_basis and there is a reference stimulus amount R_basis, the output amplitude A is calculated by the following equation (4).
A = A_basis × (R ÷ R_basis) (4)
The above example is an example of amplitude, but may be applied as a frequency. When the stimulating unit is an LED, the brightness may be adjusted by calculating the amount of current flowing through the LED in the same manner as in the above formula.
The above formula (4) is merely an example, and other relational expressions may be used. A process of converting the stimulus amount R into an appropriate value as an input value to the stimulus unit may be performed. The output of step S306 for performing signal conversion processing for the stimulus output unit 57 is data of a signal that can be input to the stimulus output unit 57 calculated in the above processing. For example, the amplitude value. For example, a frequency value. For example, the current value. For example, a voltage value. There may be one output value or a plurality of output values. For example, amplitude and frequency values may be used.

Next, the input of step S307 for performing output processing to the stimulus output unit 57 is the output of step S306, which is data that has undergone signal conversion processing for the stimulus output unit 57. For example, when the stimulus output unit 57 is a vibration motor, the value is an amplitude value. The process of step S307 for performing the output process to the stimulus output unit 57 is a process for inputting the input data to the stimulus output unit 57 and actually performing the stimulus output. For example, this is a process of inputting an amplitude value command to the vibration motor. The output of step S307 for performing output processing to the stimulus output unit 57 is that the stimulus output unit 57 is outputting. For example, the vibration motor is oscillating with the amplitude input to the vibration motor in the above processing. In the above description, the state of changing the amplitude value of the vibration motor has been specifically described. However, the current value, resistance value, and voltage may be changed in order to change the light amount and color using an LED or the like. Further, output processing may be performed such that the volume is changed using a speaker.
In this way, by generating at least one of the light wave signal, sound wave signal, vibration, and odor that can be sensed by the first participant, the degree of intention to participate in the conference that appears in the speech and behavior of the remote participant is increased. The amount of intention to represent can be presented.

<Seventh embodiment>
FIG. 9 is a diagram illustrating a specific example of the avatar robot 17 representing the second participant 15 shown in FIG. 1 used in the communication assist system according to the seventh embodiment of the present invention.
17 is an avatar robot representing the second participant 15, 17 a is a stimulating unit located in the avatar robot's ear, 17 b is a stimulating unit located in the eyes of the avatar robot, 17 c is a stimulating unit located in the mouth of the avatar robot, and 17 d is A stimulation unit located in the hand of the avatar robot, 17e is a stimulation unit located on the foot of the avatar robot, and 17f is a stimulation unit located inside the avatar robot.
For example, when the avatar robot 17 representing the second participant 15 stimulates vision, any or a part or all of the stimulation units 17a to 17f move, the color changes, or the state changes according to time. It is an avatar robot provided with such a stimulating unit.
For example, when the avatar robot 17 representing the second participant 15 stimulates hearing, a stimulating unit in which sound output from any one or a part or all of the stimulating units 17a to 17f changes according to time. It is an avatar robot provided.
For example, when the avatar robot 17 representing the second participant 15 stimulates the sense of touch, any or a part or all of the stimulation units 17a to 17f move, the state changes according to time, and the robot is touched. It is an avatar robot having a stimulation unit that can recognize the difference in state. The stimulus output unit 57 generates a vibration that can be sensed when the first participant contacts.

For example, when the avatar robot 17 representing the second participant 15 stimulates an olfaction, the odor is emitted from any or a part or all of the stimulation units 17a to 17f in accordance with a time change, and the difference in state using the olfaction It is an avatar robot provided with a stimulation unit that can recognize the The stimulus output unit 57 generates an odor that can be sensed when the first participant smells it.
For example, the avatar robot 17 representing the second participant 15 is an avatar robot in which the stimulation unit 17a located at the ear of the avatar robot is a vibration motor.
For example, the avatar robot 17 representing the second participant 15 is an avatar robot in which the stimulation unit 17b located in the eyes of the avatar robot is an LED.
For example, the avatar robot 17 representing the second participant 15 is an avatar robot in which the stimulation unit 17c located in the mouth of the avatar robot is a speaker.
For example, the avatar robot 17 representing the second participant 15 is an avatar robot in which the stimulation unit 17d located in the hand of the avatar robot is a vibration motor.
Although the above description is limited to a specific stimulation unit, the stimulation unit may be at least one of the stimulation units 17a to 17f illustrated in FIG.
In this way, by using the distance between the first participant and the robot unit 17 as the first site status information, an intention amount representing the degree of willingness to participate in the conference that appears in the behavior of the remote participant is presented. The degree can be adjusted.

<Eighth Embodiment>
10 to 13 are diagrams for explaining the distribution of the sensation E of the first base 11A reflecting the incandescence of the video conference of the first base 11A used in the communication assist system according to the eighth embodiment of the present invention. FIG.
FIG. 10 is a graph for specifically explaining a casual range of the distribution of the sense E of the first participant 13 and the third participant 14 at the first base 11A.
For example, from the measured incandescence of the conference at the first base 11A, it is assumed that the conference E has a distribution of sensation E for each conference participant, and has a range of sensations E_min to E_max.

FIG. 11A is a graph for specifically explaining a casual range of the sensation E of the first base 11A from the distribution of the sensation E of the first participant 13 and the third participant 14 of the first base 11A. It is.
For example, a range in which both the first participant 13 and the third participant 14 at the first base 11A feel casual is set as a casual range of the sense E of the first base 11A. Although the above is a value obtained by taking the AND of the participants, the casual range of the sensation E of the first base 11A may be calculated from the distribution of the sensation E of the plurality of participants by other methods. It can also be calculated by the following formulas (5) and (6).

E_max = min [E_max_13, E_max_14] (5)
E_min = max [E_min_13, E_max_14] (6)

FIG. 11 (b) calculates the sensation E that the second participant wants to give from the range where the distribution of the sensation E per participant is casual when there are many participants at the first base 11A. It is a graph for demonstrating the method to do.
For example, let us consider a case where there are many participants at the first base 11A, and there is a range where each participant feels that the distribution of the sensation E is casual as shown in FIG. For example, when the second participant wants to be aware of his / her presence is very high, a specific example is a method of calculating the value of sensation E that the participant thinks is casual. One.
For example, if the second participant wants to be aware of his / her presence is very low, he / she thinks that some people are casual, and the remaining humans have a sense E that is less than E_min. One method is to calculate the value of.

FIG. 12A is a graph for explaining a specific example of the probability density of the maximum sensation E_max of a plurality of participants who feels casually in the first base 11A.
For example, the range of the sensation E of the first base 11A can be updated using a probability density distribution as shown in FIG. For example, when the first site 11A is exciting, there is a method of adopting the E_max value that increases the probability density as the maximum value E_max of the sense that the first site 11A feels casual. Conversely, when the first site 11A is not excited, there is a method of adopting the E_max value that reduces the probability density as the maximum value E_max that the first site 11A feels casual.
FIG. 12B is a diagram illustrating a specific example of the probability density of the minimum value E_min of the sensation felt by a plurality of participants present at the first base 11A.
It is the figure explaining the minimum value of the sensation felt casually in the update of the range of the sensation E of the 1st base 11A. The specific method is the same as (a).

In FIG. 12 (c), a predetermined range E of the first base 11A that is felt casually is defined as E_max_basis and E_min_basis, and according to the base state information of the video conference of the first base 11A, It is a graph for demonstrating the specific example which updates the range E_max and E_min of the amount of sensations which one base 11A feels casually.
The predetermined range E of the first base 11A that is felt casually is defined as E_max_basis and E_min_basis. For example, when the video conference at the first base 11A is incandescent, the first base 11A is casual. The sensory amount ranges E_max and E_min that are felt to be absent are converted or calculated into large values. It is calculated using the following formula.

E_max = f (MotionValue_11A, MotionCount_11A, E_max_basis) (7)
E_min = f (MotionValue_11A, MotionCount_11A, E_min_basis) (8)
Here, f is a function that has been preliminarily set.

FIG. 13 is a specific example of calculating the sensory amount E to be presented to the first site participant, reflecting the degree to which the second participant 15 at the second site 11B wants to be aware of his / her presence. It is a graph for demonstrating.
For example, the degree that the second participant 15 wants to be aware of his / her presence is calculated by adjusting the value in the range according to the above degree in the range shown in FIG. For example, if you want to be strongly aware of your presence, take a value close to E_max. When the degree of wanting to be aware of one's presence is small, the value is close to E_min.

The contents described below may be applied to all the above embodiments.
For example, the degree to which the participant 15 at the second base 11B wants to be aware of his / her own presence has been calculated based on the action of the second participant in the embodiments so far. It may be measured and calculated with a signal such as
In addition, the calculation process of the stimulus R is the distance between the avatar robot 17 representing the second participant 15 and the first participant 13 and the third participant 14 at the first base 11A, or the avatar representing the second participant 15. Strength proportional to the distance between the robot 17 and the first participant 13 at the first base 11A, or the distance between the avatar robot 17 representing the second participant 15 and the third participant 14 at the first base 11A. It may be a process to calculate by.
Further, there may be a plurality of speakers when the site status information of the first site 11A is measured and the calculation process and the determination process are performed.
Thus, based on the second site status information and the distribution of the amount of sensation of the first participant, the degree of presenting the amount of intention representing the degree of intention to participate in the conference that appears in the speech of the remote participant is adjusted. It can be carried out.

<Examples of Embodiments and Effects of the Present Invention>
<First aspect>
The communication assistance device of this aspect is a communication assistance device that assists communication via the communication network N between the video conference terminal 10A disposed at the first base 11A and the video conference terminal 10B disposed at the second base 11B. In the first base 11A, the wireless communication unit 53 that receives the second base state information representing the estimated state of the second base 11B and the first participant 13 in the first base 11A can perceive. The amount of intention of the second participant at the second base 11B is calculated based on the sensory amount range calculation unit 55a that calculates the range of the sensory amount, the second site status information, and the range of the sensory amount of the first participant. And a stimulus output unit 57 that outputs a stimulus according to the intention amount so that the first participant can perceive the stimulus.
According to this aspect, based on the second site situation information and the range of the first participant's sense amount, the degree of intention indicating the degree of intention to participate in the conference that appears in the speech and behavior of the remote participant Adjustments can be made. That is, it is possible to adjust the degree of presenting one's presence to the other party.

<Second aspect>
The communication assistance device of this aspect includes a first site status estimation unit 25 that estimates first site status information representing the status of the first site 11A, and the intention amount calculation unit 55b includes the first site status information and the second site. The amount of intention of the second participant is calculated based on the situation information and the range of the sense amount of the first participant.
According to this aspect, the intention indicating the degree of willingness to participate in the conference that appears in the behavior of the remote participant based on the first site status information, the second site status information, and the range of the sense amount of the first participant The degree of presenting the amount can be adjusted. That is, it is possible to adjust the degree of presenting one's presence to the other party.

<Third aspect>
In the communication assisting device of this aspect, the sensory amount range calculating unit 55a calculates the distribution of sensory amounts that can be perceived by the first participant, and the participation intention calculating unit 55b includes the second site situation information and the first participant. The intention amount is calculated based on the distribution of the sense amount.
According to this aspect, based on the second site situation information and the distribution of the sense amount of the first participant, the degree of presenting the amount of intention representing the degree of intention to participate in the conference that appears in the speech of the remote participant Adjustments can be made. That is, it is possible to adjust the degree of presenting one's presence to the other party.

<4th aspect>
In the communication assisting device of this aspect, the first site situation estimation unit 25 is based on the first audio / video measurement unit that measures the audio signal and the video signal of the first site 11A and the video signal of the first site 11A. A first characteristic action extracting means for extracting one characteristic action information; a specifying means for specifying the speaker information and the conversation partner information of the speaker based on the audio signal and the video signal of the first base 11A; Determination means for determining whether or not the information represents a second participant, and the first site situation estimation unit 25 receives the first characteristic motion information, the speaker information, and the conversation partner information as the first site situation. It is output as information.
According to this aspect, by using the first characteristic operation information, the speaker information, and the conversation partner information as the first site situation information, the degree of intention to participate in the conference that appears in the behavior of the remote participant is expressed. The degree of presenting the will can be adjusted. That is, it is possible to adjust the degree of presenting one's presence to the other party.

<5th aspect>
In the communication assisting device of this aspect, the stimulus output unit 57 includes the robot unit 17 configured to output a stimulus, and the first site situation estimation unit 25 is a distance between the first participant and the robot unit 17. A distance measuring means for measuring the distance, and the distance is estimated as the first site status information.
According to this aspect, by using the distance between the first participant and the robot unit 17 as the first site situation information, the will amount representing the degree of intention to participate in the conference that appears in the speech and behavior of the remote participant. The degree of presentation can be adjusted. That is, it is possible to adjust the degree of presenting one's presence to the other party.

<Sixth aspect>
In the communication assisting device of this aspect, the stimulus output unit 57 generates a light wave signal that can be sensed by visual observation by the first participant, and a sound wave signal that can be sensed by listening to the first participant. At least of a sound wave generating means for generating, a vibration generating means for generating a perceivable vibration when the first participant contacts, and an odor generating means for generating a perceptible odor when the first participant sniffs It is characterized by being one.
According to this aspect, by generating at least one of a light wave signal, a sound wave signal, vibration, and odor that can be sensed by the first participant, the intention of participating in the conference that appears in the speech and behavior of the remote participant An intention amount representing the degree can be presented. That is, it is possible to adjust the degree of presenting one's presence to the other party.

<Seventh aspect>
In the communication assistance device of this aspect, the second site situation estimation unit 45 includes a second audio / video measurement unit that measures an audio signal and a video signal as the status of the second site 11B, and the second site 11B within a predetermined time. When the audio signal is less than or equal to a predetermined volume, the second characteristic action extracting means for extracting the second characteristic action information based on the video signal of the second place 11B, and the second characteristic action information as the second condition Transmission means for transmitting information to the first base 11A side.
According to this aspect, the second characteristic operation information is extracted based on the video signal of the second base 11B when the audio signal of the second base 11B within a predetermined time is below the predetermined volume. It is possible to calculate the amount of intention representing the degree of intention to participate in the conference that appears in the behavior of the participant.

<Eighth aspect>
In the communication assistance system according to this aspect, the communication assistance apparatus according to any one of the first to sixth aspects is used as the stimulus amount calculation output unit 50, and the communication assistance apparatus according to the seventh aspect is the second site situation estimation unit 40. It is characterized by providing as.
According to this aspect, a communication assistance system can be configured using the stimulus amount calculation output unit 50 and the second site situation estimation unit 40.

<Ninth aspect>
The communication assistance method of this aspect is a communication assistance that assists communication via the communication network N between the first conference terminal 10A arranged at the first base 11A and the second conference terminal 10B arranged at the second base 11B. In this method, the first base 11A can perceive the receiving step S200 for receiving the second base state information indicating the estimated state of the second base 11B and the first participant 13 in the first base 11A. The amount of intention of the second participant 15 at the second base 11B based on the sensory amount range calculating step S300 for calculating the range of the sensory amount, the second site status information and the range of the sensory amount of the first participant 13 An intention amount calculating step S300 for calculating the stimulus, a stimulus output step S400 for outputting the stimulus according to the intention amount so that the first participant 13 can perceive, Characterized in that it comprises.
According to this aspect, based on the second site status information and the range of the first participant's sense amount, the degree of presenting the intention amount indicating the degree of intention to participate in the conference that appears in the speech and behavior of the remote participant Adjustments can be made. That is, it is possible to adjust the degree of presenting one's presence to the other party.

<10th aspect>
The program according to this aspect is characterized by causing a processor to execute each step according to the ninth aspect.
According to this aspect, since each step can be executed by the CPU, it is possible to adjust the degree of presenting the amount of intention representing the degree of intention to participate in the conference that appears in the speech and behavior of the remote participants.

  DESCRIPTION OF SYMBOLS 4 ... Imaging device, 5 ... Microphone, 6 ... Speaker, 7 ... Display, 8 ... Processing device, 10 ... Video conference terminal, 11A ... First base, 11B ... Second base, 13 ... First participant, 14 ... First 3 participants, 15 ... second participants, 17 ... avatar robot, 20 ... first site situation measurement estimation unit, 21 ... first site situation measurement unit, 23 ... first site situation capture unit, 25 ... first site Situation estimation unit, 27 ... wireless communication unit, 30 ... relay unit, 40 ... second site status measurement estimation unit, 41 ... second site status measurement unit, 43 ... second site status capture unit, 45 ... second site status Estimator 50 ... Stimulus amount calculation output unit 55 ... Stimulus amount calculation unit 55a ... Sense amount range calculation unit 55b ... Intention amount calculation unit 55c ... Stimulus amount calculation unit 57 ... Stimulus output unit N ... Communication network

Patent No. 521101

http: // WWW. oki. com / jp / otr / 2008 / n213 / pdf / 213_r06. pdf

Claims (10)

A communication assistance device for assisting communication via a network between a first conference terminal arranged at a first site and a second conference terminal arranged at a second site,
In the first base,
Receiving means for receiving second site status information representing the estimated status of the second site;
A sensory amount range calculating means for calculating a range of a sensory amount perceivable by the first participant at the first base;
An intention amount calculating means for calculating an intention amount of the second participant at the second base based on the second base situation information and the range of the sensory amount of the first participant;
And a stimulus output means for outputting a stimulus according to the amount of intention so that the first participant can perceive the communication assist device. First estimation means for estimating first site status information representing the status of the first site,
The intention amount calculating means includes:
2. The communication according to claim 1, wherein the amount of intention of the second participant is calculated based on the first location status information, the second location status information, and the range of the sensory amount of the first participant. Auxiliary device. The sensory amount range calculating means calculates a distribution of sensory amounts perceivable by the first participant,
The communication assistance device according to claim 1, wherein the intention amount calculating unit calculates the intention amount based on the second site situation information and a distribution of a sense amount of the first participant. The first estimating means includes
First audio and video measurement means for measuring the audio signal and video signal of the first base;
First characteristic action extracting means for extracting first characteristic action information based on the video signal of the first base;
Identifying means for identifying speaker information and conversation partner information of the speaker based on the audio signal and video signal of the first site;
Determination means for determining whether or not the conversation partner information represents the second participant,
The first estimating means includes
The communication assisting apparatus according to claim 1, wherein the first characteristic operation information, the speaker information, and the conversation partner information are output as the first site status information. The stimulus output means includes
A robot unit configured to output the stimulus,
The first estimating means includes
Ranging means for measuring the distance between the first participant and the robot unit,
The communication assisting apparatus according to claim 1, wherein the distance is estimated as the first site status information. The stimulus output means includes
A light wave generating means for generating a light wave signal that can be sensed by visual observation of the first participant;
Sound wave generating means for generating a sound wave signal that can be sensed by listening to the first participant
Vibration generating means for generating a perceptible vibration when the first participant contacts;
The communication assisting device according to claim 1, wherein the communication assisting device is at least one of odor generating means for generating a odor that can be sensed by smelling the first participant. The second estimating means includes
A second audio / video measuring means for measuring an audio signal and a video signal as the status of the second base;
Second characteristic motion extraction means for extracting second characteristic motion information based on the video signal of the second site when the audio signal of the second site within a predetermined time is below a predetermined volume;
The communication auxiliary device according to claim 1, further comprising: a transmission unit configured to transmit the second characteristic operation information to the first site as the second site status information. The communication auxiliary device according to any one of claims 1 to 6 is a first unit,
A communication support system comprising the communication support device according to claim 7 as a second unit. A communication assistance method for assisting communication via a network between a first conference terminal arranged at a first site and a second conference terminal arranged at a second site,
In the first base,
A receiving step of receiving second site status information representing the estimated status of the second site;
A sensory amount range calculating step of calculating a range of a sensory amount perceivable by the first participant at the first base;
An intention amount calculating step for calculating an intention amount of the second participant at the second base based on the second base situation information and the range of the sensory amount of the first participant;
And a stimulus output step of outputting a stimulus according to the amount of intention so that the first participant can perceive.   A program for causing a processor to execute each step according to claim 9.

Images