JPH07184180A - Conference environment controller - Google Patents

Abstract

PURPOSE:To attain the conference in a sense close to a face-to-face conference by controlling a video signal and an audio signal in an assumed virtual conference space in a multi-point video conference system and applying abstract processing to a request from the user as an operation in the virtual conference space. CONSTITUTION:An environmental information storage processing means 11, an environmental information notice means 12, a line connection control means 13 are used to implement management of conference environmental information assuming a virtual conference space and connection control based on the management. A terminal equipment control means 21 and a user request processing means 22 are used to attain video and audio signal control for a terminal equipment based on the environmental information and the entry of the user request abstracted as the operation in a virtual conference room. Thus, the users make communication under an environment closer to a face-to-face environment as the virtual conference room.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multipoint video conference in which a group of video conference terminals at distant points are mutually connected to perform a communication conference, connection switching between video conference terminals and images at each video conference terminal. The present invention relates to a conference environment control device that controls a voice environment.

[0002]

2. Description of the Related Art Conventionally, an MCU (Multipoint Control) has been used as a control device for a multipoint video conference system.
olUnit) is used. FIG. 9 shows a system configuration of a multipoint video conference using an MCU when the number of points is 4. In a system using an MCU, video conference terminals are arranged at points A, B, C and D, and these terminals are connected to the MCU via a line. The MCU has a function of transmitting video and audio signals received from each terminal through a line to an appropriate terminal by appropriate connection control.

For the audio signal, for example, a signal obtained by mixing the sounds of all the terminals B, C, D except A is sent to the terminal A, and for the video signal, B, C,
One of the images D is transmitted. As described above, when the number of terminals participating in the conference increases, it becomes impossible, for example, to send all the images of the other points B, C, and D to the terminal A at the same time. It is necessary to send the data to the other side, or to combine the B, C, and D images and convert them into a format that can be sent. In the method of switching, manual switching or automatic switching by automatic speaker detection is used. On the other hand, when performing image synthesis, a simple screen division method or a combination of a large screen and a small screen display method is used.

Further, as a development of this, as shown in Japanese Patent Laid-Open No. 1-280982, relational information indicating the relative position between the image and the sound image is set, and the image / sound image control is performed based on the relation information. The scheme is known.

[0005]

Processing such as connection switching, screen switching, and screen composition between the MCU and the terminal is mechanically performed depending on the physical structure of the system. As a result, the operation of the user and the method of displaying the screen depend on physical elements such as the connection relationship of the line, and the user has to have the conference in an environment significantly different from the face-to-face conference environment. . This is a factor that impedes smooth dialogue in video conferences. In addition, in the system disclosed in Japanese Patent Laid-Open No. 1-280982,
It is possible to control each terminal assuming an independent environment,
The environment of the entire system is not managed.
Therefore, if there is a change in the environment, it is not possible to reflect it in the environment of each terminal, and it is not possible to manage the environment consistently as a whole system, and due to environmental changes etc. However, there is a drawback in that it cannot be dealt with in the case of occurrence of. The present invention has been made in view of this point, and by consistently performing connection switching and terminal control assuming a virtual conference room in the entire system,
It is intended to provide a video conference environment with little difference from the face-to-face conference even when the assumed environment changes.

[0006]

In order to achieve the above-mentioned object, a conference environment control apparatus of the present invention has a video conference terminal capable of inputting / outputting video and audio, and a television set installed at a plurality of three or more locations. In a multipoint video conferencing system consisting of a centralized video conferencing control device that controls line connection switching between conference terminals, accumulating and managing conference environment information assuming a virtual conference room and responding to requests from users. Environment information storage processing means for updating the meeting environment information,
Centralized control device having line connection control means for switching connection based on meeting environment information and environment information notifying means for transmitting the contents to the terminals when the meeting environment information is updated, and centralized control Terminal control means that receives the conference environment information sent from the device and controls the video and audio output in the terminal based on it, interprets the request from the user as operation in the virtual conference room and sends the content to the centralized control device It is characterized by comprising a terminal device provided with a user request processing means.

[0007]

In the multipoint video conference system using the conference environment control device according to the present invention, the environmental information storage processing means for managing the environmental information of the virtual conference space, the environmental information notifying means for transmitting the environmental information to the terminal, and A user request process that builds a virtual meeting space with a realistic presence in front of the user by means of a terminal control unit that displays video and outputs audio based on environmental information, and that receives an operation request from the user as an operation in the virtual space. By the means, the user can hold a conference in a virtual conference space with a feeling similar to a face-to-face conference.

[0008]

Embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram of a conference environment control device of the present invention. The multipoint video conference system according to the present invention includes a centralized control device 1 and a terminal device 2 arranged at a place where a conference is held. The centralized control device 1 and the terminal device 2 are connected via a line, and the centralized control device 1 is also connected to other terminal devices (not shown) by the same means as 2. Furthermore, in the multipoint video conference system according to the present invention, a system using a video / audio signal other than NTSC can be constructed.

The centralized control device 1 includes a line connection switching device 15 for switching connection between terminals for transmitting and receiving a video signal and an audio signal to each other, a line interface 14 for connecting to a line, and a virtual interface. Environmental information accumulation processing means 11 having a function of accumulating and managing the environmental information of the conference space and updating the environmental information in response to a request from the terminal device 2,
It is composed of a line connection control means 13 for controlling the line connection switching device 15 based on the environment information and an environment information notifying means 12 for transmitting the contents to the terminal device 2 when the environment information is updated.

The terminal device 2 is a line interface 2 for connecting to a line for communicating video and audio signals.
4, the video / audio input / output device 23 for displaying and outputting the received video / audio signal and inputting the video / audio signal to be transmitted, and the video / audio input / output device 23 is controlled based on the environmental information transmitted from the centralized control device 1. And a user request processing unit 22 for inputting a request from a user as an operation in the virtual conference space.

Next, FIG. 2 is a block diagram showing the configuration of each part in the first embodiment of the present invention. The configuration of each part in this embodiment will be described.

The environment information storage processing means 11 creates a meeting environment database 113 that stores information about a typical face-to-face meeting environment, which is a model of a virtual meeting space, and a model most suitable for the content of the meeting from the meeting environment database 113. Environmental information managing means 112 for searching and managing environmental information of the virtual conference space based on the information, and user request receiving means 11 for transmitting a request from the terminal to the environmental information managing means 112.
1 and.

The environment information notifying means 12 comprises an environment information transmitting means 121 for transmitting the contents to the terminal when there is a change in the environment information.

The line connection control means 13 includes an environment information conversion means 132 for converting the environment information of the virtual conference space into physical connection information, and a line information database for storing the line information necessary for the conversion. 133 and switching device control means 1 for controlling the line connection switching device 15 based on the connection information obtained by the environment information conversion means 132.
31 and 31.

The terminal control means 21 receives the environment information sent from the centralized control device 1 when the environment information changes, and the environment information receiving means 213 for determining the setting of the terminal from the received environment information. Terminal environment construction means 212
And terminal device control means 211 for controlling the terminal based on the terminal information obtained there.

The user request processing means 22 is a user request input means 221 for inputting a request from a user as an operation in the virtual conference space, and a user request transmitting means for transmitting the request to the centralized control device 1. It is composed of 222.

FIG. 3 is a diagram showing a first embodiment of the data structure managed in the environment information management means 112 and the meeting environment database 113. The management of the virtual conference environment will be described with reference to FIGS. 2 and 3. The virtual conference environment is composed of static environment information that can be defined in advance according to the form of the conference and dynamic environment information that changes as the conference progresses. The static management information is stored in the conference environment database 113, and the dynamic environment information is managed by the environment information management means 112. Virtual conference space models 301a, 301b, 3 in FIG.
Reference numeral 01c is the first embodiment of the static environment information and represents a typical face-to-face meeting form such as a round table format, a face-to-face format, and a lecture format. Each model is defined with the maximum number of seats N and the positional relationship between seats as elements, and the conference environment database 113
Stored in. Inside the conference environment database 113, the positional relationship between seats is managed in the form of a dialogue rule table 310. The dialogue rule table includes a user request received via the user request receiving unit 111 of FIG.
It stores the change rules of the seat management table 320 for each request. Also, the dialogue rule table 310
Is created for each typical example of the defined meeting form, and the illustrated interaction rule table 310 illustrates a part of the rules corresponding to the round table form 301a.

The seat management table 320 in FIG.
5 is a first example of dynamic environment information managed by the environment information management means 112. The dynamic environment information in this example includes a seat number, a terminal name virtually arranged in each seat, a virtual line-of-sight direction from each seat, and a line-of-sight distance that means the distance from each seat to an object. It consists of one item. When the environmental information management unit 112 receives a conference holding request from the user, the environmental information management unit 112 retrieves static environmental information suitable for the content of the conference from the conference environment database 113, and the seat management table 320.
To the initial state. Furthermore, when a request from the user (for example, a line-of-sight movement request) is received, the environmental information management means 1
12 applies the processing rule stored in the dialogue rule table 310 to update the seat management table 320, and transfers the updated environment information to the environment information conversion means 132.

FIG. 4 shows a first embodiment of the data structure managed by the environment information conversion means 132. Management of physical environment information will be described with reference to FIGS. 2, 3, and 4. The physical environment information represents the connection relationship between the input line and the output line in the centralized control device 1. The physical environment information in this example includes line information indicating the connection relationship between the line connection switching device 15 and each terminal, and connection management information indicating the state of the line connection switching device 15 at that time. The line information is stored in the line information database 133 in the form of an input line management table 402 and an output line management table 403. The connection management information is the connection management table 40.
The environment information conversion unit 132 manages the data in the format 1. Here, the connection management table 401 shows the state of connection between the input line and the output line by an adjacency matrix, and for example, 0 indicates not connected and 1 indicates connection.

The environment information conversion means 132 receives the environment information from the environment information management means 112 when there is a change in the virtual conference environment. As shown in the seat management table 320 of FIG. 3, the virtual conference environment information includes the line-of-sight direction from each seat and the name of a terminal virtually arranged in each seat. Here, in the seat management table 320, when the line-of-sight direction from the terminal A is the seat 3 and the terminal name of the seat 3 is the terminal C, the image of the terminal C is displayed on the image of the terminal A. Means to display on the device. That is, the environment information of the virtual conference space represents connection relation information of video and audio signals between terminals. Also, an input line management table 402 and an output line management table 403
Shows the correspondence between the line number in the line connection switching device 15 and each terminal connected thereto. Therefore, the environment information conversion unit 132 uses the environment information of the virtual conference space and the line information 40 stored in the line information database 133.
2, 403 to refer to the line connection switching device 1
The connection at 5 can be determined. The environment information conversion unit 132 reflects the determined contents on the connection management table 401, and further, the contents are connected device control unit 131.
And the connection is changed in the line connection switching device 15. At the same time, the environment information conversion unit 132 receives the contents of the seat table 320 received from the environment information management unit 112, the contents of the terminal-by-terminal line information table 410, and FIG.
The form type and the maximum number of seats N in the virtual conference space model 301 are transmitted to each terminal via the environment information transmitting means 121. Here, the terminal-by-terminal line information table 410 represents, by a terminal name, the contents flowing to each line when each terminal has a plurality of receiving lines.

FIG. 5 shows the video / audio input / output device 2 in FIG.
FIG. 3 is a diagram illustrating an implementation according to the first example of Example 3. The video / audio input / output device 23 is the line interface 24 of FIG.
A video / audio separation means 231 for separating a plurality of video / audio signals received via a video signal and an audio signal, respectively.
a, 231b, 231c, a window video synthesizing device 235 for synthesizing the separated video signal and the background video stored in the background video storage device 234 by window display based on an instruction from the terminal device control means 211, A video display device 237 for displaying the captured video, 231
The audio localization control means 232a, 232b, which can convert the audio signals separated by a, 231b, 231c into stereo audio in a state where the audio is localized at a desired position based on an instruction from the terminal device control means 211. Two
32c, and a sound mixing device 233a, 2 that mixes a plurality of generated stereo sounds into one left sound and one right sound.
33b and a speaker 2 for outputting mixed sound
36a, 236b, and a camera 239 and a microphone 238 for transmitting the user's video and audio. Here, the display windows 237a, 237b, 23
7c is arranged on the image display device 237 in accordance with the seat arrangement in the virtual conference space, and further, the voice uttered by the person displayed in each display window is adjusted to the position of each display window on the screen. Video and audio are output in a state that matches the seat arrangement in the virtual conference space. At the same time, a background image having contents suitable for the virtual conference space is fetched from the background image storage device 234 and displayed as the background of the image display device 237, so that the user can be more aware of the virtual conference space. It will be possible.

FIG. 6 shows the video / audio input / output device 2 shown in FIG.
FIG. 3 is a diagram for explaining the implementation of the third example according to the second example. The video / audio input / output device 23 includes video display devices 252a and 252b arranged on the left and right and a speaker 23 attached thereto.
6a, 236b, a selection device 251 for selectively connecting a required line from a plurality of lines based on an instruction from the terminal device control means 211, and a video signal and a voice signal separated from each other. It is composed of video / audio separating means 231a and 231b to be sent to the display device and the speaker, and a camera 239 and a microphone 238 for transmitting the video and audio of the user. In this example, the terminal device control means 2
In the instruction from 11 to the selection device 251, by issuing an instruction in which the left-right positional relationship between the video display device and the speaker and the positional relationship of the seats in the virtual conference space are associated with each other, the virtual conference space is created as in the first embodiment. The effect of making the user aware can be obtained.

In the virtual conference space 711 and screen setting 712 in FIG. 7, it is assumed that the six terminals A, B, C, D, E, and F hold a conference in the form of a round table 301a in FIG. The state of the screen display in the virtual conference space and the terminal A is shown. The terminal management table 701 is the first embodiment of the data structure used for terminal management in the terminal environment construction means 212 of FIG. Hereinafter, the processing in the terminal environment construction means 212 will be described with reference to FIGS. 3, 4, 5, and 7. The terminal environment construction means 212, via the environment information receiving means 213, the contents of the seat management table 320 and the terminal-specific line information table 41.
The contents of 0, the form type and the maximum number of seats N in the virtual conference space model 301 of FIG. 3 are received. The terminal environment construction means 212 can construct a virtual conference space as indicated by 711 using these pieces of information. Here, the terminal environment construction means 212 has information on the physical structure of the terminal, determines the screen setting of the terminal based on the information of the virtual conference space and the physical structure information of the terminal constructed earlier, and displays the result. It is stored in the terminal management table 701. For example, when the terminal has the structure shown in FIG. 5 and the virtual conference space is in the state of 711, the terminal environment construction means 212 determines to make the screen setting shown in 712, and the result is the terminal management table 701. To store. Further, the terminal environment construction means 212
Displays the contents of the terminal management table 701 in the terminal device control means 2
Hand over to 11. In the terminal device control means 211, the contents of the terminal management table 701 are stored in the window video composition device 23.
5 and voice localization control devices 232a, 232b, 232
It is converted into a concrete operation command to c and controls those devices. Through the above processing, the terminal device can control video and audio according to the environment of the virtual conference space.

User request input means 221 in FIG.
Is for receiving a request from a user.
The request that can be accepted is expressed as the action of the participant in the virtual conference space, and the request that can be accepted is determined according to the form of the virtual conference space. For example, in the case of the round table format 301a in FIG. 3, the dialogue rule table 3
The user request registered in 10 can be accepted. The user request and its specific effect will be described below with reference to FIG. FIG. 8 is a diagram for explaining a state change of a virtual conference space and an actual screen display according to a user request. Here, when a conference is held by connecting six terminals A, B, C, D, E, and F in the round-table-type virtual conference space 301a shown in FIG. Suppose state I (801) is heading to D. In this state, the user "move the line of sight to the left"
The following shows how the environmental information in the virtual space changes and how the screen display is controlled on terminal A when four requests are made: "move gaze to the right,""move gaze closer," and "move gaze away." There is. However, the terminal A is assumed to be the terminal according to the first embodiment shown in FIG. Also,
State diagrams 801, 802, 803, 804, 80 in the figure
5, the virtual conference space 810 represents the concrete state of the virtual conference space model 301a in FIG.
Reference numeral 1 represents a display state on the screen display device 237 of FIG. A state II (802) shows a state change due to a user request having the content of “move the line of sight to the left”. The line of sight of terminal A in the virtual space moves to the left, and as a result,
From the left, the contents of the video display on terminal A are terminals C, D,
What is displayed as E is changed so that the images on the left for one seat are displayed to the terminals B, C, and D. State II
I (803) indicates a state change due to a user request having the content of “move eyes to the right”. The line of sight of the terminal A moves to the right in the virtual space, and as a result, the contents of the image display on the terminal A are the terminals C, D, and E displayed from the left to the terminals D, E, and F. It is changing so that the image on the right is displayed. The state IV (804) shows a state change due to a user request of the content "to bring the line of sight closer". The line-of-sight distance of the terminal A in the virtual space becomes short, and as a result, the content of the video display on the terminal A is
What is displayed from the left on the terminals C, D, and E is changed to only the enlarged image of the terminal D. A state V (805) indicates a state change due to a user request having the content of "keep the line of sight away". The line-of-sight distance of the terminal A in the virtual space becomes long, and as a result, the contents of the video display on the terminal A are
The display of the terminals C, D, and E from the left has changed to the display of the reduced image of the terminals B, C, D, E, and F with a wider field of view. Thus, the request from the user is interpreted as a virtual operation in the virtual conference space. Then, the contents of the environmental information changed with it are transmitted to the terminal,
Further, by controlling the terminal device in accordance with the environmental information, the user's request is reflected on the screen according to the rules in the virtual conference space.

[0025]

As described above in detail, according to the present invention, in the multipoint video conference system, the environmental information storage processing means 11 for managing the environmental information of the virtual conference space and the environmental information notification for transmitting the environmental information to the terminal. The means 12 and the terminal control means 21 for displaying the image and outputting the sound based on the environment information construct a virtual conference space in front of the user with a realistic feeling, and further, as an operation in the virtual space, an operation request from the user. The user request processing means 22 for accepting the request enables the user to hold a meeting in the virtual meeting space with a feeling similar to a face-to-face meeting.

[Brief description of drawings]

FIG. 1 is a block diagram of a conference environment control device.

FIG. 2 is a block diagram showing an embodiment of a conference environment control device.

FIG. 3 is a diagram showing an example of a data structure for managing environment information of a virtual conference space.

FIG. 4 is a diagram showing an example of a data structure for managing physical connection information in the centralized control device.

FIG. 5 is an explanatory diagram of implementation of the video / audio input / output device according to the first embodiment.

FIG. 6 is an illustration of implementation of the second embodiment of the video / audio input / output device.

FIG. 7 is a diagram illustrating processing in the terminal control device.

FIG. 8 is a diagram illustrating a relationship between a user request and a screen control method in a round table-style conference.

FIG. 9 is a configuration diagram of a video conference system using a conventional MCU.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Centralized control device 2 Terminal device 11 Environmental information storage processing means 12 Environmental information notification means 13 Line connection control means 14 Line interface 15 Line connection switching device 21 Terminal control means 22 User request processing means 23 Video / audio input / output device 24 Line interface 111 user request receiving means 112 environment information managing means 113 conference environment database 121 environment information transmitting means 131 switching device control device 132 environment information converting means 133 line information database 211 terminal device controlling means 212 terminal environment constructing means 213 environment information receiving means 221 User request input means 222 User request transmission means 231a, 231b, 231c Video / audio separation means 232a, 232b, 232c Audio localization control means 233a, 233b Audio mixing device 234 Background video storage device 35 window image synthesizer 236a, 236b speaker 237 image display device 237a, 237b, 237c display window 238 microphone 239 camera 251 selection device 252a, 252b image display device 301 virtual conference space model 310 dialogue rule table 320 seat management table 401 connection management table 402 Input round management table 403 Output round management table 410 Terminal-based line information table 701 Terminal management table 711 Virtual conference space 712 Screen setting 801 State I 802 State II 803 State III 804 State IV 805 State V 810 Virtual conference space 811 Video display device

Claims (1)

[Claims] 1. A video conferencing terminal capable of inputting / outputting video and audio, and a centralized video conferencing control device for controlling line connection switching between video conferencing terminals installed at three or more points. In a multi-point video conferencing system that is used, an environment information storage processing unit that stores and manages the meeting environment information assuming a virtual meeting room and updates the meeting environment information in response to a request from the user, and based on the meeting environment information A centralized control device having line connection control means for switching the connection, and environmental information notification means for transmitting the contents of the conference environment information to each terminal when the environmental information is updated; The terminal control means for receiving the meeting environment information and controlling the video / audio output in the terminal based on the information, and interpreting the request from the user as the operation in the virtual meeting room Conferencing environment control device, characterized in that it is configured from a terminal and a user request processing unit for transmitting to the central control device.