JP4496060B2 - Transition support method and video / audio communication system - Google Patents

Description

  The present invention relates to a video / audio communication system that facilitates communication with a remote party, particularly when two states, that is, a work state in a real space mainly composed of personal work and a conversation state with a remote partner can be mixed. The present invention relates to a transition support method that supports smooth transition between a work state and a conversation state, and a video / audio communication system that realizes such a transition support method.

  As a tool to communicate with a remote party, obtain a camera that shoots the user, a display that displays the partner's image, a speaker that outputs the voice of the partner, and the voice of the user. There is a video / audio communication system using a microphone. In these video / audio communication systems, it is common to install a camera and a display in the horizontal direction, in other words, to install a camera and a display in front of each user.

  As a method of using such a video / audio communication system, a one-to-one remote conference has been well known so far, but a physical connection with the other party is maintained, but a conversation with the other party is always performed. Rather, there is a usage in which a conversation with the other party is performed intermittently (or occasionally) and personal work (for example, the user's own desk work) is performed in real time at other times. In other words, multiple people are doing desk work in one room, and each person is usually doing their own personal work, but when a question or confirmation occurs, Talking is generally equivalent to an extended form of work to remote parties.

In this way, there are several examples of research on group work systems that do not always talk with remote partners, but maintain physical connection with remote partners while mainly doing personal work in real space. Can be given. In these research examples, it is supported that the conversation using the video / audio communication system is started by presenting the situation information of the other party to the user in advance. Research on systems that can acquire each other's situation by always connecting video (Non-Patent Document 1), and research on systems that can obtain the other party's situation by presenting the proximity information of the remote party as a visual effect (Non-Patent Document 1) (Patent Document 2), research on a system that can walk around the virtual world and start a conversation while looking at the remote partner's desk (Non-Patent Document 3), presenting information using the user's location information and schedule information System research (Non-patent Document 4) is being conducted. However, these are not considering where to present the situation information, so there is a possibility that it will be presented in a place that hinders the user's personal work, and the user who wants to concentrate differs due to the transition from the work state to the conversation state It does not consider changes in characteristics.
"Evaluating Video as a Technology for Informal Communication", RS Fish, CHI '92 "The Effects of Visual Proxemic Information in Video Mediated Communication", D. Grayson, L. Coventry, CHI '98 "Design of a Multi Media Vehicle for Social Browsing", RW Root, Proceedings of the 1988 ACM conference on CSCW "Context Aware Messaging Service: Construction and operation experiment of mopile communication system using location information and schedule information", Yasuhito Nakanishi, Takataka Tsuji, Minoru Oyama, Katsuya Hakozaki, IPSJ Transactions, Vol. 37, No . 6

  In a conventional video / audio communication system, when a display used for conversation is installed in front of a user, the display is presented to the user who is working personally. The situation on the side will be distracted and distracted, which may hinder the user's work. Also, when the user is about to start a conversation or during a conversation, he / she wants to work to see if the face direction of the remote party is the face direction where he / she is looking at the user through the display. It is difficult to determine whether the face direction is working. For this reason, there is a problem that the smooth start of the conversation is hindered, for example, it is difficult to talk to the other party or it is difficult to sense the desire of the other party to talk.

  Accordingly, an object of the present invention is not to hinder a user's work in a video / audio communication system used in an environment in which two states of a work state in a real space mainly for personal work and a conversation state with a remote partner are mixed. The transition support method that can present the other party's video in such a way that the user can concentrate on the work during the work, and can smoothly switch to the conversation state with the other party when the user wants to talk with the other party And a video / audio communication system for realizing such a transition support method.

  The transition support method of the present invention uses a first display arranged in front of a user and a second display arranged in a direction different from the first display, and receives video and audio from a remote partner. A transition support method for supporting a transition between a work state in a real space mainly for a user's personal work and a conversation state with a remote partner in a video / audio communication system to be presented to a user. Detecting the face direction, detecting no-speech time in the conversation with the user and the remote partner, detecting that the face direction is directed to the second display, and transitioning to the conversation mode; And a step of shifting to a work mode when the face direction is not directed to any of the displays and the non-speech time continues for a certain time or more.

  The video / audio communication system of the present invention is used in an environment in which a work state in a real space mainly for a user's personal work and a conversation state with a remote partner are mixed, and video and audio from the remote partner are used for the user. A video / audio communication system to be presented, a first display arranged in a front direction of a user, a second display arranged in a direction different from the first display, and arranged in a front direction of the user The first camera for photographing the user, the second camera for photographing the user arranged at a different position from the first camera, the microphone for recording the user's speech, and the speech of the remote partner A speaker that reproduces the sound toward the user, a face direction detection unit that detects the face direction of the user based on the images taken by the first and second cameras, and no speech based on the sound input to the microphone time Based on the data of the non-speech time measurement unit to be measured, the face direction detection unit, and the non-speech time measurement unit, it is determined that the face direction is directed to the second display, and the face direction is determined Is not facing the direction of any display, and if no talk time continues for a certain time or more, it is determined that the mode has been changed to the work mode, and the remote partner's video is displayed on each display according to the determination result. And a mode determination unit for controlling.

  The present invention obstructs the user's work by utilizing the user's face direction and the state of the utterance in an environment where two states of a work state in a real space mainly for personal work and a conversation state with a remote partner are mixed. By selecting the display screen of the other party's video that should not be used, it is possible to concentrate on the work during the user's work, and to smoothly switch to the conversation with the other person when the user wants to talk.

  Next, a preferred embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a video / audio communication system according to an embodiment of the present invention.

  The video / audio communication system is arranged on the user 100 side and the partner 200 side, respectively, and the video / audio communication system is connected by a network 190. It is assumed that a desk 150 for the user 100 is provided on the user 100 side, and a desk 250 for the person is provided on the partner 200 side.

  The video / audio communication system of the present embodiment has two modes, a conversation mode and a work mode. The conversation mode is a state in which the user 100 is talking with the partner 200, wants to talk with the partner 200, or is watching the situation of the partner 200 on the display to have a conversation (a conversation state with a remote partner). ) Mode. On the other hand, in the work mode, the user 100 does not have a conversation with the partner 200 and concentrates on the personal work of the user 100 (work state in a real space mainly composed of personal work). Is a mode corresponding to. As will be described below, the video / audio communication system detects the state of the user 100, makes a transition between the conversation mode and the work mode, and changes the content presented on the display according to the determined mode.

  The video / audio communication system on the user 100 side has a screen facing the user 100 who is sitting on a chair of the desk 150 and a ceiling display 101 installed on the ceiling above the desk 150; The ceiling camera 102 installed on the ceiling above the desk 150 with the optical axis directed to the head of the user 100 sitting on the chair of the desk 150 and the front position of the user 100 sitting on the chair of the desk 150 The desk display 103 arranged, the desk camera 104 installed with the directional axis directed to the head of the user 100 sitting on the chair of the desk 150, the microphone 105 that records the utterance of the user 100, and the ceiling A face direction detection unit that detects the face direction of the user 100 based on an image captured by the camera 102 and / or the desk camera 104. 06, a non-speech time measuring unit 107 that measures a non-speech time in a conversation between the user 100 and the other party 200 based on the voice input by the microphone 105, a face direction detecting unit 106, and a non-speech time measuring unit 107 Based on the detection or measurement result, a mode determination unit 108 that determines whether the mode is a conversation mode or a work mode, a communication control unit 109 that controls communication via the network 190 to the video / audio communication system on the partner 200 side, And a speaker 110 that reproduces the utterance of the partner 200 at the user 100 toward the user 100. The desk display 103 is installed on the desk 150 or in front of the desk 150, for example. As the desk display 103, a display of a personal computer used by the user 100 for the personal work can be used.

  Similarly, the video / audio communication system on the partner 200 side also includes a ceiling display 201, a ceiling camera 202, a desk display 203, a desk camera 204, a microphone 205, a face direction detection unit 206, and a speechless time measurement unit 207. A mode determination unit 208, a communication control unit 209, and a speaker 210.

  In the video / audio communication system of the user 100, the image of the partner 200 captured by the ceiling camera 202 on the partner 200 side is displayed on the ceiling display 101 when at least the face of the user 100 faces the direction of the ceiling display 101. Presented. In addition, the desk display 103 presents an image of the partner 200 captured by the desk camera 204 on the partner 200 side in the conversation mode. Even in the video / audio communication system of the other party 200, the image of the user 100 captured by the ceiling camera 102 on the user 100 side is displayed on the ceiling display 201 when at least the face of the other party 200 faces the direction of the ceiling display 201. When presented and the mode determination unit 208 determines the conversation mode, the desk display 103 presents the video of the user 100 taken by the desk camera 104 on the user 100 side.

  Next, the operation of this video / audio communication system will be described.

  First, mode determination by the mode determination unit 108 will be described.

  The face direction detection unit 106 extracts the face area of the user 100 from the image captured by the ceiling camera 102 and / or the desk camera 104 by detecting the skin color area, and also detects the area of the eyes and nose of the user 100 in the area. The position is detected, and the face direction of the user 100 is determined from these results. The non-speech time measuring unit 107 measures the time when the user 100 and the partner 200 do not speak each other based on the voice acquired by the microphone 105. In any mode, the mode determination unit 108 wants the user 100 to have a conversation with the partner 200 or the partner 200 when the face direction of the user 100 faces the direction of the ceiling display 101. It is determined that the situation of the partner 200 is being viewed in order to talk to the user, and the conversation mode is entered. When the mode is changed to the conversation mode, when the face of the user 100 faces the ceiling display 101, the video of the partner 200 acquired by the ceiling camera 202 on the partner 200 side is presented on the ceiling display 101, and the face of the user 100 is displayed. When facing the desk display 103, the display that is presented according to the face direction of the user 100, such as presenting the video of the partner 200 acquired by the desk camera 204 on the partner 200 side on the desk display 103. To decide. The mode determination unit 108 is in the conversation mode, the face direction of the user 100 is not facing any of the displays 101 and 103, and the no-speaking time is longer than a certain time (for example, 3 seconds), The user 100 is considered to be in a state where he / she concentrates on the individual work without talking to the presented partner 200, and transitions to the work mode. In the work mode, the mode determination unit 108 presents the partner video only on the ceiling display 101 so as not to disturb the work of the user 100.

  As described above, in the present embodiment, the mode determination unit 108 (or the mode determination unit 208) can automatically switch between the conversation mode and the work mode. You can concentrate on your work and you can smoothly switch from work to conversation with your partner.

  FIG. 2 is a flowchart showing processing in the video / audio communication system on the user 100 side. As a matter of course, the video / audio communication system on the partner 200 side performs the same process.

  Assuming that the initial state is the work mode, in step 301, the operation mode is entered. In the work mode, when the user 100 is working on the desk 150 regardless of whether or not the desk display 103 is used, the ceiling display 101 receives the data transmitted from the ceiling camera 202 on the partner 200 side. Present (step 302). The desk display 103 presents work data or does not present anything. Since the work mode is a state in which the other users are not trying to communicate with each other, no sound is input from the microphone 105 and the other party's sound from the speaker 110 is not reproduced. In the work mode, the mode determination unit 108 makes a determination based on the face direction data of the user 100 detected by the face direction detection unit 106 based on the images taken by the ceiling camera 102 and the desk camera 104 (step 303). Only when the face direction of the user is the direction of the ceiling display 101, the process proceeds to step 304 and transitions to the conversation mode. Otherwise, the process returns to step 301.

  In the conversation mode, the user's face direction is always determined, and when the face direction of the user 100 is the direction of the ceiling display 101, the data transmitted from the ceiling camera 202 on the partner 200 side is presented to the ceiling display 101, When the face direction is the desk display 103 direction, the data transmitted from the desk camera 204 on the partner 200 side is presented on the desk display 103. When the face direction is a direction other than those, the presentation on the display previously presented is continued. That is, the video is presented following the face direction (step 305). Then, in step 306, the mode determination unit 107 first determines whether or not the non-speech time is a predetermined time or more based on the data of the non-speech time measurement unit 107. Return to step 304 to continue the mode. If the non-speaking time is equal to or longer than a certain time, the mode determination unit 108 determines in step 307 whether the face direction of the user is a direction other than the display 101 or 103 direction based on the data of the face direction detection unit 106. If the direction is other than the display direction, the user 100 considers that the user 100 is not conversing with the partner 200 and concentrates on the individual work, and proceeds to step 301 to shift to the work mode. If the face direction is the display direction, the process returns to step 304 as continuing the conversation mode.

  In this way, the mode determination unit 108 executes the mode determination process.

  Next, a display method on the desk display 103 will be described. When the video of the desk camera 204 of the other party 200 is displayed on the desk display 103 in the conversation mode, when the user 100 is reading and not using the desk display 103 for work, the desk display 103 Try to present the video on the entire surface. On the other hand, when the user 100 is using the desk display 103 for work such as sending and receiving e-mails, the user first searches for a portion without character display on the display, and an area without character display exists more than a certain size. Then, the video is presented in that area. In the case where such an area is inside, the video is presented at a place where there are many areas without character display among the four corners of the display surface of the desk display 103.

  In the video / audio communication system described above, the face direction detection unit 106, the non-speech time measurement unit 107, the mode determination unit 108, and the communication control unit 109 store a computer program for realizing the above in a computer such as a personal computer. It can also be realized by reading and executing the program. Such a program is read into a computer by a recording medium such as a CD-ROM or via a network. Such a computer is generally connected to a CPU (Central Processing Unit), a hard disk device for storing programs and data, a main memory, input devices such as a keyboard and a mouse, and a ceiling display 101 and a desk display 103. A video adapter, a reading device that reads a recording medium such as a CD-ROM, an interface unit to which the ceiling camera 102, the desk camera 104, the microphone 105, the speaker 110, and the like are connected, and a communication for connecting to the network 190 Interface.

It is a block diagram which shows the structure of the video / audio communication system of one Embodiment of this invention. It is a flowchart explaining the mode transition in the audio video communication system shown in FIG.

Explanation of symbols

100 User 101, 201 Ceiling display 102, 202 Ceiling camera 103, 203 Desk display 104, 204 Desk camera 105, 205 Microphone 106, 206 Face direction detection unit 107, 207 Speechless time measurement unit 108, 208 Mode determination unit 109, 209 Communication control unit 110, 210 Speaker 150, 250 Desk 190 Network 200

Claims (7)

Video and audio presenting video and audio from a remote partner to the user using a first display arranged in front of the user and a second display arranged in a direction different from the first display A transition support method for supporting transition between a working state in a real space mainly for a user's personal work in a communication system and a conversation state with a remote partner,
Detecting a face direction of the user;
Detecting speechless time in a conversation with the user and the remote party;
Detecting that the face direction is directed to the second display and transitioning to a conversation mode;
When the face direction does not face the direction of any display, and when the no-speech time continues for a certain time or more, a transition to a work mode;
A transition support method. In the conversation mode, in the conversation mode, the video from the front of the remote partner is presented by the first display, and the video from other than the front of the remote partner is presented by the second display. Following the face direction and presenting the image of the remote party on the first and second displays;
In the work mode, stopping the presentation of the image of the remote partner on the first display;
The transition support method according to claim 1, further comprising:   The user is photographed by two different cameras, the skin color area detection is performed on the photographed video, the face area of the user is extracted, and the user's eyes and nose are extracted in the extracted area. The transition support method according to claim 1, wherein a face direction of the user is determined by detecting a position. A video / audio communication system that is used in an environment where a work state in a real space mainly for a user's personal work and a conversation state with a remote partner are mixed, and presents video and audio from the remote partner to the user. There,
A first display arranged in the front direction of the user;
A second display arranged in a different direction from the first display;
A first camera arranged in a front direction of the user to photograph the user;
A second camera arranged at a position different from the first camera and photographing the user;
A microphone that records the user's utterance;
A speaker for reproducing the utterance of the remote partner toward the user;
A face direction detection unit for detecting the face direction of the user based on images taken by the first and second cameras;
A non-speech time measuring unit that measures the non-speech time based on the sound input to the microphone;
Based on the data of the face direction detecting unit and the non-speech time measuring unit, it is determined that the face direction is directed to the second display and it is determined that the conversation mode has been changed, and the face direction is any display. If the no-speaking time has continued for a certain time or more, it is determined that the mode has shifted to the work mode, and the display of the remote partner's video is displayed on each display according to the determination result. A mode determination unit to control;
A video / audio communication system.   In the conversation mode, the mode determination unit presents an image from the front of the remote partner on the first display, and presents an image from other than the front of the remote partner on the second display. The video of the remote partner is presented by the first and second displays following the face direction, and the presentation of the video of the remote partner by the first display is stopped in the work mode. The audiovisual communication system according to 1.   The video / audio communication system according to claim 4, wherein the second display and the second camera are provided on a ceiling in the overhead direction of the user. Video and audio presenting video and audio from a remote partner to the user using a first display arranged in front of the user and a second display arranged in a direction different from the first display In the computer constituting the communication system,
Processing to detect the user's face direction;
A process for detecting a speechless time in a conversation between the user and the remote party;
A process of detecting that the face direction is directed to the second display and transitioning to a conversation mode;
A process of shifting to a work mode when the face direction is not directed to any display and the no-speaking time has continued for a certain period of time;
In the conversation mode, the image from the front of the remote partner is presented on the first display, and the image from the front of the remote partner is presented on the second display, following the face direction. A process of presenting the image of the remote party on the first and second displays;
In the work mode, a process of stopping the presentation of the remote partner's video on the first display;
And a program to execute.

Images