JP2000217091A - Video conference system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video conference system by which progress of a more natural and smoother video conference can be realized. SOLUTION: In this video conference system, video images of conference participants of other terminals are side by side displayed on a long sideways screen of a video output device 17, a plurality of video cameras 11a-11z are respectively placed above the video images of the respective participants displayed on the long sideways screen. Non-verbal interpretation processing is applied to the motion of each conference participant on the basis of a plurality of video information sets obtained by photographing a talker in different directions by using the video cameras 11a-11z so as to specify a person to which the talker speaks thereby attaining control such as automatic transfer of a right to speak from the talker. Thus, the progress of the video conference can be realized more naturally and smoothly.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] The present invention relates to a video conference system.

[0002]

2. Description of the Related Art In recent years, terminals located at remote locations, such as offices and branches, are connected via a communication network so that conference participants at each location can stay at each location. A video conference system for performing a video conference by data communication has been realized.

As an example of a terminal installed at each site of a conventional video conference system, as shown in FIG. 5, an image of a conference participant from another terminal installed at another site is displayed on a screen. There is a monitor having a monitor 51 that is divided and displayed, and a single camera 52 that is installed on the monitor 51 and captures a conference participant of the own terminal that talks while watching the screen of the monitor 51.

Since this type of video conference system uses hardware equivalent to a general personal computer (hereinafter referred to as a personal computer), the monitor 51 has a diagonal size of, for example, 15 inches to 19 inches. The screen of the monitor 51 is used. Therefore, when the conference is performed by about five people including yourself, the screen of the monitor 51 includes:
The images of the other conference participants are displayed in four parts, one frame at a time, vertically and horizontally.

[0005] In this case, the viewpoint of the conference participant of the own terminal watching the screen of the monitor 51 is always within the range of the screen of the monitor 51,
That is, it is restricted to one point.

[0006]

However, in this, non-verbal communication with a conversation partner important in the progress of a conference, such as movement of eyes, changes in pupil size, changes in facial expressions, changes in gestures and hand gestures, and the like, are required. There was a problem that the conversation was unnatural, especially in multi-person meetings.

SUMMARY OF THE INVENTION The present invention has been made to solve such a problem, and the present invention is directed to a dialogue operation performed by a speaker, for example, movement of a line of sight, change in pupil size, change in facial expression, change in gesture,
A feature of the present invention is to provide a video conference system that allows a video conference to proceed smoothly in a more natural manner by considering non-verbal communication such as hand gestures.

[0008]

In order to achieve the above-mentioned object, a video conference system according to claim 1 establishes a communication network between a terminal and a plurality of other terminals located at different points. In the video conference system connected via the terminal, each terminal, when at least one of the conference participants of the own terminal speaks to the video of the plurality of conference participants sent from the other terminal Display means for arranging the participant so as to change the interaction with others, and arranged at the video position of each conference participant displayed by the display means or at a position near each video, and the conference participant of the terminal itself. A plurality of photographing means for photographing, the processing device on each terminal or the communication network, based on a plurality of video information respectively photographed by the plurality of photographing means,
Non-verbal interpretation processing means for performing non-verbal interpretation processing and identifying an operation in which the speaker selects one of the plurality of conference participants and speaks is provided.

According to a second aspect of the present invention, there is provided a video conference system in which a terminal is connected to a plurality of other terminals arranged at different points via a communication network. Display means for displaying the images of the plurality of conference participants sent from the other terminal in such a manner that at least when the conference participant of the own terminal speaks toward one of them, it changes the interaction operation with the others. And a plurality of photographing means arranged at a video position of each conference participant displayed by the display means or a position near each video, and a plurality of photographing means for photographing the conference participant of the own terminal, The processing device on the communication network performs a non-linguistic interpretation process based on the plurality of pieces of video information respectively captured by the plurality of capturing units, and the speaker determines one of the plurality of conference participants. A non-verbal interpretation processing means for identifying an action of selecting and speaking; and a non-verbal interpretation processing result by the non-verbal interpretation processing means, for delegating the speaking right to the terminal of the identified one conference participant. Control means for performing switching control.

According to a third aspect of the present invention, in the video conference system according to any one of the first and second aspects, the display means displays the images of the plurality of conference participants in order in a horizontal direction. Has a screen.

According to the first aspect of the present invention, at least when the conference participant of the own terminal speaks toward one of the plurality of conference participants transmitted from the other terminal, an interactive operation with the other is performed. Display means for displaying side by side to change the
A plurality of photographing means for photographing the conference participant of the own terminal are arranged at the image position of each conference participant displayed by the display means or at a position near each image, so that each terminal or communication A processing device on the network performs a non-linguistic interpretation process based on a plurality of pieces of video information respectively captured by a plurality of capturing means, and an operation in which a speaker selects one of a plurality of conference participants and speaks. , It is possible to know which conference participant the speaker is speaking to, and to perform various controls for smoother conference management.

According to the second aspect of the present invention, in addition to the first aspect of the present invention, a speaker selects a video from a plurality of conference participants and speaks, and automatically speaks to the person. Since the control of transferring the right is performed, the degree of freedom of the non-verbal communication is increased and the speaker can speak naturally.

According to the third aspect of the present invention, the display means has a screen for displaying images of a plurality of conference participants in order in the horizontal direction, so that at least the speaker turns his / her eyes to the conversation partner. Changes in behavior can occur and meetings with nonverbal communication can be held.

That is, according to the present invention, the video of the conference participant of the other terminal is displayed side by side, and a plurality of photographing means are installed at the position of the video of each conference participant. Non-verbal interpretation processing of the speaker's actions for each of a plurality of video information obtained from each photographing means by photographing the speaker from different directions, and specifying the person with whom the speaker interacted, for example, to give the speaker the right to speak Since control such as automatic transfer to the interlocutor can be performed, a more natural and smooth progress of the video conference can be realized.

[0015]

Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a diagram showing one embodiment of a video conference system according to the present invention.

As shown in FIG. 1, this video conference system includes video conference terminals 1a to 1z arranged at a plurality of bases dispersed in various places, and these terminals 1a to 1z.
1z via a communication network 2 (MC)
U) 3 constitute a system. The central processing unit (MCU) 3 includes video conference terminals 1a to 1 at each base.
The entire conference is controlled, for example, by transmitting video data and the like transmitted from z to designated sites and executing processing based on control data from each site. The terminals 1a to 1z for the video conference at each base and the central processing unit (MCU) 3 are connected to the central processing unit (MCU) 3 in a communication network 2 in, for example, a star type.

Each terminal 1a to 1z for a video conference is shown in FIG.
, A plurality of video cameras 11a to 11z,
Video information captured by the video cameras 11a to 11z is input, the input video information is subjected to non-verbal interpretation processing, and a speaker selects one of a plurality of conference participants and speaks. A video input processing device 12 for identifying a video of a conversation partner from an operation, an audio input processing device 14 connected to a microphone 13, a transmission processing device 15 for multiplexing and transmitting each data, and a reception process for receiving and separating each data A video output processing device 18 connected to a device 16, a video output device 17 such as a large-screen liquid crystal projector having a horizontally long screen (landscape screen), an audio output processing device 20 connected to a speaker 19, and the entire terminal And a control device 21 for controlling the Video cameras 11a-1
1z outputs the video signal of the conference participant at the base to the video input processing device 12. Video input processing device 12
Performs non-verbal interpretation processing, compression encoding processing, distribution destination addition processing, and the like of each video input signal input from the video cameras 11a to 11z, and the configuration will be described later. The audio signal from the microphone 13 is output to the audio input processing device 14. The voice input processing device 14 performs a compression encoding process and the like. The transmission processing device 15 multiplexes and transmits signals from the video input processing device 12, the audio input processing device 14, and the control device 21. The reception processing device 16 receives the multiplexed video, audio, and control signals, performs a separation process on these signals, and outputs a video output processing device 18,
It is transmitted to the audio output processing device 20 and the control device 21.
The video output processing device 18 decodes the compressed data and reconstructs the video of each of the other sites on the video output device 17. The speaker 19 generates sound. The audio output processing device 20 decodes the compressed data and outputs the decoded data to the speaker 9. The control device 21 controls the entire site.

As shown in FIG. 3, the video input processing device 12 includes an analog-to-digital converter (hereinafter referred to as an ADC).
24a to 24z, video compression / encoding processing unit 25a
To 25z, non-language interpretation processing units 26a to 26z, and the like.

The ADCs 24a to 24z are the video cameras 1
Video signal (analog signal) input from 1a to 11z
Is converted into a digital signal. The video compression encoding processing units 25a to 25z are encoders that compress and encode moving image video data. Non-linguistic interpretation processing units 26a to 26
z performs non-verbal interpretation processing on video input from the ADCs 24a to 24z based on gaze movement, pupil size, facial expression, gesture, hand gesture, and the like, and transmits the result to the control device 21. The control device 21 sends information (non-language interpretation data) from the non-language interpretation processing units 26a to 26z to the transmission processing device 15, and the transmission processing device 15 transmits the non-language interpretation data to the central processing unit (MCU) 3. I do. The central processing unit (MCU) 3 performs transfer control of the right to speak based on the movement of the line of sight based on the non-linguistic interpretation data transmitted from the terminal at each site, that is, control for switching speakers in a conference and control for promoting communication. Do. In addition, for example, translation of non-verbal communication between different cultures may be performed.

The video output device 17 provided in each of the terminals 1a to 1z for the video conference, as shown in FIG.
In the case of images of a plurality of conference participants transmitted from terminals other than the own terminal, for example, images of four participants, the images are displayed in each of a plurality of areas 41a to 41d divided in the horizontal direction of the landscape screen 40. On the upper surface of the video output device 17 above each of the regions 41a to 41d, there are provided a plurality of video cameras 11a to 11e to shoot the upper body part of the conference participant of the own terminal to be transmitted to each conference participant displayed in each of the regions 41a to 41d. 11d is arranged close to the position of each conference participant image.

Next, a specific operation of the video conference system will be described. In this video conference system, for example, a video conference is held between a conference participant of the own terminal and four conference participants of a plurality of other terminals, and the conference participant of the own terminal has a floor as a speaker. When the speaker speaks, for example, to the person at the right corner of the landscape screen 40 in FIG. 4, that is, the conference participant displayed in the area 41d, the line of sight of the speaker is Since the speaker is directed in the direction of 41d, the image of the speaker captured by the video camera 11d is an image of the face in front of the speaker. In particular, the direction in which the speaker's eyes are facing, that is, the line of sight is directed to the front. ing.

At this time, another video camera 11a
11c are images of the face other than the front of the speaker, that is, a profile or oblique profile.

The video information is input to the ADCs 24a to 24z in the form of video signals (analog signals) and converted into digital signals.
5a to 25z and output to the non-language interpretation processing units 26a to 26z.

In the video compression / encoding processing sections 25a to 25z, the input digital signal is
2 or H. The video data is compression-encoded into moving image data such as H.261 and output to the transmission processing device 15.

In addition, the non-verbal interpretation processing units 26a to 26z respond to the input of video information from the ADCs 24a to 24z from changes in gaze movement, changes in pupil size, changes in facial expressions, gestures, changes in hand gestures, and the like. Perform a non-linguistic interpretation,
It interprets which video camera the speaker is talking to in the images obtained from the plurality of video cameras 11a to 11d.

In this case, the non-language interpretation processing unit 26d includes:
The video of the front face of the speaker is input as the video of the video camera 11d to which the line of sight is directed from the speaker, and the non-verbal interpretation processing unit 26b outputs the profile of the speaker as the video from the video camera 11b. The video is input, and the video of the speaker's oblique side profile is input to the non-language interpretation processing unit 26c as the video from the video camera 11c.

From these images, the direction information of each face and the like are output to the control device 21 as respective interpretation results.

The control device 21 determines from the results of these interpretations that the conference participant shown in the area 41d where the video camera 11d is installed is the talker with the current speaker. The result of this determination is that the video number of the interlocutor is used as data of the result of the non-language interpretation.
Sent to

The transmission processing device 15 transmits the non-verbal interpretation result data to the central processing unit (MCU) 3 together with the input moving image video data.

The central processing unit (MCU) 3 performs the transfer control of the right to speak, that is, the switching of the talker of the conference, based on the data of the non-language interpretation result transmitted from the terminal at each base. The conference participant shown in 41d is allowed to speak and can speak. In addition, communication facilitation control for presenting “please say” to the interlocutor may be performed.

As described above, the speaker in front of the terminal can select a person who wants to have a conversation by moving his / her gaze to the conference participant displayed on the screen or changing the face direction as in an actual conference. The degree of freedom of non-verbal communication is expanded, and it is possible to speak naturally.

As described above, according to the video conference system of this embodiment, the video of the conference participant of another terminal is displayed side by side on a horizontally long monitor, and a plurality of video are displayed at the positions of the video of each conference participant. Video cameras 11a to 11
By setting z, a speaker is photographed from each direction, and a plurality of video information obtained from each of the video cameras 11a to 11z is subjected to non-verbal interpretation processing of the motion of the speaker to specify a conversation partner on the system side. Therefore, a more natural and smooth progress of the video conference can be realized.

The present invention is not limited only to the above embodiment. In the above embodiment, the video cameras are arranged by the number corresponding to the video of the conference participant. However, when there are only two conference participants other than the conference participant of the own terminal, the conference participation of the other terminal is performed. Each image of the user is displayed on two different monitors arranged at a predetermined interval, and even if one camera is arranged between the two monitors, the user speaks in the direction of one of the images. Sufficient video information can be obtained to identify the action of the speaker.

In the above embodiment, one large-screen liquid crystal projector having a horizontally long screen 40 has been described as an example. In addition, for example, monitors for the number of conference participants are prepared and arranged at predetermined intervals. But it is good.

In order to conduct a video conference more realistically, a plurality of speakers are prepared around the conference participant of each terminal, and a three-dimensional sound field is given to the conference participant by the acoustic effect of the plurality of speakers. May be formed so that the listener can recognize who the speaker is.

[0036]

As described above, according to the present invention, images of conference participants at other terminals are displayed side by side, and a plurality of photographing means are installed at the positions of the images of each conference participant. By photographing the speaker from different directions with the respective photographing means, performing non-verbal interpretation processing of the speaker's actions for each of a plurality of video information obtained from each photographing means, and specifying the person with whom the speaker interacted, For example, control such as automatically transferring the right to speak from the speaker to the interlocutor can be performed, and a more natural and smooth progress of the video conference can be realized.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a video conference system according to an embodiment of the present invention.

FIG. 2 is an exemplary block diagram showing an example of terminals installed at the base of the video conference system.

FIG. 3 is a diagram showing a configuration of a video input processing device of the video conference system.

FIG. 4 is a diagram showing a specific arrangement example of video cameras in the video conference system.

FIG. 5 is a diagram showing an example of the arrangement of video cameras in a conventional video conference system.

[Explanation of symbols]

11a to 11z video camera, 12 video input processing device, 13 microphone, 14 audio input processing device, 15 transmission processing device, 16 reception processing device, 17 video output device, 18 video output processing device 19 ... speaker, 20
... Sound output processing device, 21 ... Control device, 1a to 1z ... Terminal, 2 ... Communication network, 3 ... Central processing unit (MCU), 24a
２４24z ADC, 25a〜25z video compression coding processing unit, 26a〜26z non-language interpretation processing unit.

Claims (3)

[Claims] 1. In a video conference system in which a terminal and a plurality of other terminals arranged at different points are connected via a communication network, each terminal is transmitted from the other terminal. Display means for displaying images of a plurality of conference participants, at least when the conference participant of the own terminal speaks toward one, and changes the interactive operation with the other, and displayed by the display means. A plurality of photographing means arranged respectively at the video position of each conference participant or at a position near each video, and photographing the conference participant of the own terminal, wherein each terminal or the processing device on the communication network comprises: Performing a non-linguistic interpretation process based on a plurality of pieces of video information respectively captured by the plurality of capturing units, and identifying an operation in which the speaker selects one of the plurality of conference participants and speaks; Video conference system, characterized by comprising a language interpretation process means. 2. In a video conference system in which the terminal itself and a plurality of other terminals arranged at different points from the terminal are connected via a communication network, each terminal is transmitted from the other terminal. Display means for displaying images of a plurality of conference participants, at least when the conference participant of the own terminal speaks toward one, and changes the interactive operation with the other, and displayed by the display means. A plurality of photographing means arranged respectively at the video position of each conference participant or at a position near each video, and photographing the conference participant of the own terminal, wherein each terminal or the processing device on the communication network comprises: Performing a non-linguistic interpretation process based on a plurality of pieces of video information respectively captured by the plurality of capturing units, and identifying an operation in which the speaker selects one of the plurality of conference participants and speaks; Language interpretation processing means, and control means for performing switching control for delegating the speaking right to the terminal of one identified conference participant as a result of the non-language interpretation processing by the non-language interpretation processing means. A video conference system characterized by the following. 3. The video conference system according to claim 1, wherein the display unit has a screen for displaying images of the plurality of conference participants in order in a horizontal direction. Video conference system.