JPH0730877A - Inter-multi location multimedia communications conference system - Google Patents

Abstract

PURPOSE:To attain communication of persons taking part in the conference by generating notice information representing who of conference participants is to be noticed and displaying the information on its own multi-medium communications conference equipment. CONSTITUTION:A visual line information transmission display section 6 gives notice party data inputted from the user and position data to a transmission section 7. The visual line information transmission display section 6 receives position data of other participants and notice opposite data from a reception section 8. Furthermore, the visual line information transmission display section 6 calculates where each participant is located and whom each participant is observing based on the data and displays the result. Then the transmission section 7 adds sender destination information to picture data and sends the result to each system through a transmission line 30. Moreover, the transmission section 7 adds sender and destination information to the position data and the notice opposite data obtained from the visual line information transmission display section 6 and sends the result to each system. The information of data kind is added in this transmission.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multipoint multimedia communication conference system, which is formed by at least multimedia communication between two points.

[0002]

2. Description of the Related Art In recent years, many systems for moving image communication have been put into practical use due to improvements in image processing technology and network technology. One of them is the video conference system, which ranges from floor-mounted type with a dedicated cabinet to desktop type using a personal computer or workstation as a communication terminal.

In addition, not only connecting two points but also connecting three or more points have been developed.

For example, the following documents are available for the purpose of realizing such a video conference system.

Literature: IEICE Technical Report IE
90-49, issued in 1990, "Multipoint multimedia communication conference system PMTC for B-ISDN" and the like.

And, such a video conference system is
Basically, each participant's face image is transmitted and the meeting is recommended while watching the other person's appearance. In particular, in a multipoint video conference system, face images of a plurality of partners are displayed at the same time, and a conference is held while watching the appearance of all participants. This state is shown in FIG.

However, in order to display a plurality of images, for example, as shown in FIG.
However, there is a problem that the scale of hardware becomes large.

Now, the video conference system of FIG. 3 will be described. This video conference system includes three decoders 4. In FIG. 3, the encoder 1 is means for encoding the image data input from the camera. The transmission unit 2 is a unit that adds information such as a transmission source and a transmission destination to the encoded image data and transmits the image data. Then, the receiving unit 3 receives the data from the other system, divides the data into image data and other information, further classifies the data according to the transmission source, and the decoders 4a to 4a.
It is a means to assign to c.

The decoder 4 shown in FIG. 3 is a means for decoding the image data, and a plurality of decoders 4 are provided. When the number of transmission sources is close to the number of decoders, one decoder decodes a plurality of images. In that case, each of the decoders 4a to 4c requires a memory for the number of images to be decoded. The image display means 5 is means for displaying the image data from the respective decoders 4a to 4c.

The face image generally displayed is a bust of each participant captured from the front, and the facial expression, complexion, clothes, surroundings, etc. of the participant can be seen. However, since the user does not always talk while looking at the camera, his or her line of sight may face another direction, and in this case, the lines of sight do not match.

[0011] In general, when a person generally wants to strongly assert his / her opinion to the other person, he / she speaks to the other person with his / her eyes. Furthermore, those who are seen will recognize that they are talking to themselves and will listen to them more seriously than usual. Therefore, the technique of matching the line of sight is important, but that is not enough information about the line of sight.

FIG. 4 is an example showing the direction of the line of sight in an actual conference room. The circles ○ indicate participants. Furthermore, the arrow ↑ marks indicate the direction of each line of sight. Speaker A
Is arguing for Opponent D. In response to this, the partner D looks at the speaker A, so that the speaker A and the partner D are facing each other. The line of sight is important between the two.

However, it is important for the other participants B and C that the speaker A is looking at the partner D, and the line-of-sight matching is rather an obstacle. That is, it is possible to judge that the speaker A is not talking to himself when the speaker A is not facing him.

That is, the line-of-sight information (information indicating who is watching who is talking) is one of the important information for smooth communication.

[0015]

However, in the conventional multipoint video conference system as described above, the front image of the participant is merely transmitted and displayed, and the line-of-sight information is not transmitted at all. Not done. In other words, if you want to assert (convey) an opinion to a specific person among the participants, talk to the camera (or you have completed the eye-gaze matching technology) and gaze at all the other participants. There was a problem that other people would misunderstand that they were talking in their direction.

[0016] Further, as a person who is closely watched,
There could have been a problem in which I misjudged that I wasn't speaking to me and didn't take the speaker's speech seriously as if I were speaking to me.

Therefore, from the above-mentioned problems, there is a demand for a mechanism that allows a plurality of people to hold a meeting as if they were actually facing each other in the same meeting room (room).
In other words, by adding a system (mechanism) that clearly shows the line-of-sight information of each participant, it is required to know a mechanism where each participant is currently watching and support a smooth communication.

The present invention has been made in view of the above problems, and an object of the present invention is to support communication of people participating in a conference between multiple points more easily than before. A point-to-point multimedia communication conference system is provided.

[0019]

In order to achieve the above object, a first invention transmits an image of a conference participant of its own multimedia communication conference device to another multimedia communication conference device to participate in the conference. The multi-point multimedia communication conferencing system, which displays the image of a person on its own multimedia communication conferencing device and performs multimedia communication between multi-points, has been realized with the following characteristic configuration.

In other words, the gaze information indicating which of the other conference participants the conference participant of the own multimedia communication conference device or who of the conference participants of the other multimedia communication conference device observes. Gaze information, position information, etc.), and gaze information generation means for transmitting the gaze information to another multimedia communication conference device, and gaze information display means for displaying the gaze information on the own multimedia communication conference device. Is characterized by.

Also, in the second invention, the image of the conference participant of the own multimedia communication conference device is transmitted to another multimedia communication conference device, and the image of the conference participant is displayed on the own multimedia communication conference device. The multi-point multimedia communication conference system for multi-point multi-media communication has the following characteristic configuration.

That is, the own multimedia communication conference device receives the gaze information (for example, line-of-sight information, position information, etc.) indicating who is to be gazed from another multimedia communication conference device, and displays this gaze information. It is characterized in that it is provided with a gaze information receiving and displaying means.

Further, a multipoint multi-media multimedia communication conference system having a combination of the above-mentioned first invention and second invention is also preferable.

[0024]

According to the first aspect of the invention, the gaze information generating means has a plurality of conference participants (A,
B) When there is, for example, when the conference participant A gazes at B and speaks, gaze information (eg gaze line information) from A to B is generated, and another multimedia communication conference device is generated. To be transmitted. Then, the gaze information display means can simultaneously display the gaze information on the own multimedia communication conference apparatus. Moreover, the images of the conference participants can be displayed in association with each other.

This relation may be anything. For example, the image of the participant may be a still face image or a symbol image, and the image of the participant may be displayed toward the image of the participant who is gazing at the arrow ↑ or the like.

When a conference participant A of his own multimedia communication conference device gazes at a conference participant C of another multimedia communication conference device and talks with him, the gaze information generating means changes from A to C. The gaze information of the above is transmitted to another multimedia communication conference device. Even in such a gaze state from A to C, the relationship between the conference participants can be objectively displayed as an image by the gaze information of the gaze information display means.

According to the second aspect of the invention, the own multimedia communication conference apparatus is characterized by the gaze information receiving and displaying means.
Since the gaze information can be received from other multimedia communication conferencing devices and the gaze information can be displayed on the own multimedia communication conferencing device, it is possible to objectively visually recognize who is the gaze partner among the conference participants on the image. be able to. Moreover, it can be associated with the images of the conference participants.

The relationship between the gaze information and the images of the conference participants may be any as described above. For example, the participant's image may be a moving face image, and the two images in the gaze relationship and the image of the participant not in the gaze relationship may be separately arranged.

Furthermore, by combining the structure of the above-mentioned first invention and the structure of the above-mentioned second invention,
It is possible to support communication between multiple points more and more.

In addition, (1) The above-mentioned first invention and the second invention
In the invention described above, it is more effective by adopting the following configuration. That is, for example, by generating a plan view of the composition of the conference room viewed from above, displaying the position of each participant with a circle, etc., and displaying the line-of-sight direction with an arrow, line, or dot extending from the circle, It is also possible to clearly indicate the line of sight of the participants.

(2) Further, for example, by arranging the face images and portrait images of the conference participants, the position of each participant can be displayed as an image. (3) Furthermore, for example, the line-of-sight direction can be expressed by switching and displaying the face images and portrait images facing each direction.

(4) Furthermore, for example, the position and the line-of-sight direction can be represented by synthesizing and displaying a face facing each direction by computer graphics (CG). (5)
In addition, for example, an image of a conference room with a composition viewed from one's own position is generated, a face image and a portrait image are arranged, and the face images are combined by computer graphics to represent the position and line-of-sight direction of each participant. You can also do it.

(6) Furthermore, for example, the background and the positions of other participants can be changed and displayed according to one's line of sight. (7) Furthermore, for example, the images of the participants can be switched to display only some of the participants.

(8) Further, for example, it is possible to provide a means for switching the display image according to the direction of the line of sight of the user.
(9) Furthermore, for example, a means for clearly indicating which participant's image is being displayed can be provided.

[0035]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A preferred embodiment of the present invention applied to a multipoint video conference system will be described with reference to the drawings.

(Outline of Embodiment): In this embodiment, in addition to the conventional face image data, in particular, the sight line information is transmitted and displayed on the screen. Therefore, common coordinates are prepared assuming that all the participants are gathered in one virtual place (meeting room), and the position of each participant is represented by the coordinates. Then, the participants are configured so that they can move freely on these coordinates.

Then, each participant selects the person he / she wants to see. Then, the system transmits its own gaze partner data (data indicating who he / she is looking at) and his / her position data to all the participants, as well as the position / gaze transmitted from other systems. It is configured to display the line-of-sight information of all participants on the screen using the partner data. Therefore, based on the above-mentioned purpose, the multipoint video conference system will be specifically described.

(System Configuration): FIG. 1 is a functional configuration diagram of the multipoint video conference system of the first embodiment. In FIG. 1, a multipoint video conference apparatus 20 is provided.
Is composed of an encoder 1, a transmission unit 7, an image display unit 5, decoders 4a to 4c, a reception unit 8, and a line-of-sight information transmission display unit 6. Here, the characteristic configuration is that the line-of-sight information transmission display unit 6 is provided. The transmission unit 7 and the reception unit 8 are improved in order to newly include the line-of-sight information transmission display unit 6.

The line-of-sight information transmission / display section 6 of FIG.
The gaze partner data and the position data input by the user are given to the transmitting unit 7. Then, the line-of-sight information transmission display unit 6 receives the position data of other participants and the gaze partner data from the receiving unit 8. Furthermore, where each participant is located and who is watching is calculated from these data, and this is displayed.

Then, the transmitter 7 of FIG. 1 adds the source / destination information to the image data and transmits the image data to each system through the transmission path 30. In addition, the transmitting unit 7 adds the source / destination information to the position data obtained from the line-of-sight information transmission / display unit 6 and the gaze partner data, and transmits the data to each system. At the time of this transmission, the information of the type of data (for example, image, position, watching partner) is also added.

The receiving means 8 in FIG. 1 receives the data sent from each system, gives the image data to the decoders 4a to 4c, and the position data and the gaze partner data to the visual line information transmission / display means 6. It is classified and sent.

(Structure of line-of-sight information transmission / display section 6): FIG.
FIG. 3 is a diagram showing an example of the configuration of a line-of-sight information transmission display unit 6. In FIG. 5, the line-of-sight information transmission display unit 6 includes a gaze partner selection unit 9, a position coordinate changing unit 10, a position / gaze partner data multiplexing unit 11, a gaze direction calculation unit 12, and a gaze information display unit 13. It consists of and.

Then, the gazing partner selecting section 9 of FIG.
It is to select which one of the participants of the conference is to be seen or to be seen. For example, it can be realized by directly inputting the partner name from an input device such as a keyboard. It can also be realized by designating the partner image displayed on the screen with a mouse or the like, or by designating the partner in the conference room displayed by the line-of-sight information display unit 13.

The position coordinate changing section 10 in FIG. 5 designates or changes its own position on the common coordinates in the virtual conference room. Then, the initial position can be input at the start of the conference and then changed to an arbitrary position. For example, it can be realized by directly inputting the coordinates from an input device such as a keyboard, or can be realized by designating the location of the conference room displayed on the screen by the line-of-sight information display unit 13 with a mouse or the like. Can be done.

The position / gaze partner data multiplexing unit 11 in FIG. 5 multiplexes the gaze partner data and the position data obtained by the gaze partner selecting unit 9 and the position coordinate changing unit 10. For example, as shown in FIG. 6, header information including an identifier and a participant name is added to the position data and the gaze partner data and multiplexed.

The line-of-sight direction calculation unit 12 in FIG. 5 calculates the line-of-sight direction from the position / gaze partner data of each participant including himself. For example, when participants A and B are located on common coordinates as shown in FIG. 7,
When A is looking at B, the line-of-sight direction of A can be calculated by being represented by an angle θ with respect to the x-axis.

The line-of-sight information display unit 13 of FIG. 5 displays the position of the user and other participants and the line-of-sight direction data so that it is possible to clearly indicate who is looking at which part. For example, as shown in FIG. 8, participants A to D represented by circles and bars |
Can be expressed by placing. Then, the bar line | represents the line-of-sight direction. In addition to the bar line |, as shown in FIG.

Further, as shown in FIG. 10, by arranging a face image and a portrait in the conference room, position information is represented, and by rotating these or switching to images facing different directions, the line-of-sight direction can be changed. It can also be represented.

Computer graphics (CG)
11 is used to display the state of the conference room in which the user's direction is three-dimensionally viewed, as shown in FIG.
If the display is changed in real time, a more realistic feeling can be obtained. Then, if the faces of people in the conference room are also synthesized based on the data of the person, it becomes easier to understand who is looking where.

As shown in FIG. 12, a diagram of such a conference room is displayed on the screen of the conventional system as the face images 121a to 121d.
By displaying together with, it is possible to realize a video conference system to which the line-of-sight information display 122 is added.

Originally, human beings do not see so clearly except the person they are seeing, and feel only the atmosphere. Therefore, when the line-of-sight information display becomes realistic as shown in FIG. 11, the information of other people can be obtained from here, so that it is not necessary to display the face image. That is, since only the image of the person who wants to see needs to be transmitted and displayed, for example, as shown in the functional block diagram of FIG. 13 of the second embodiment,
In this case, only one decoder 4 is required, and the scale of hardware is reduced.

In FIG. 13, the configuration is the same as that of FIG. 1 except that only one decoder 4 is provided. Therefore, at this time, the image displayed by one decoder 4 is selected by the line-of-sight direction calculation unit 12 in FIG. 5, whereby the gaze target selection unit 9 in FIG. 5 switches the image data.

If the image is requested only to the sender of the image to be viewed and the image data is not transmitted from the other participants, the load on the network is considerably reduced.

FIG. 14 of the third embodiment shows an example of the structure of the image switching type video conference system. In FIG. 14, by clearly indicating which participant's image in the conference room, it is configured so that it is easy to see who is currently watching.

(Effects of Embodiment): (1) According to the multipoint video conference system of the first to third embodiments, it is possible to objectively know who each participant is watching. ,
For example, it is possible to recognize to whom the speaker wants to particularly make an opinion, and it is possible to support smooth communication.

(2) In addition, according to the third embodiment, all the images of each participant are not transmitted and displayed at one time, and even if they are switched and displayed, the conference can be conducted without reducing the amount of information obtained. Therefore, the load on the transmission line (network) can be reduced.

(Other Embodiments): (1) In the above first embodiment shown in FIG. 1, the functional configuration is not limited to that shown in FIG. For example, although the decoders 4a to 4c are used, it is desirable that the number of decoders 4a to 4c be increased according to the number of video conference devices forming the network. Also,
Further, another functional configuration may be additionally configured. For example, a multimedia system capable of transmitting audio information and displaying the images of the above-described embodiments along with the participant's conversational audio is preferred. In addition, similarly for FIG. 13 of the second embodiment,
The configuration is not limited to this.

(2) The functional configuration of the line-of-sight information transmission / display means 6 of FIG. 5 is not limited to such a configuration. For example, another functional configuration may be added.

(3) Further, the position of FIG.
The structure of multiplexing the gaze partner data is not limited to such a structure. Other information may be additionally configured.

(4) Furthermore, the method of calculating the line-of-sight direction in FIG. 7 of the above embodiment is not limited to this.

(5) Further, FIGS. 8 to 1 of the above-mentioned embodiment.
The image display examples in FIGS. 2 and 14 are not limited to these image display examples. For example, various display methods can be applied by using a moving image, a still image, or an image synthesizing technique without changing the gist of the present invention.

(6) Furthermore, in the above-mentioned embodiment, the conference by the television was explained as an example, but the present invention is not limited to such a television system.

[0063]

As described above, according to the multipoint multimedia communication conference system of the present invention, the gaze state between the conference participants is transmitted as the gaze information so that the conference participants of the multimedia communication conference device can communicate with each other. Since the gaze information can be displayed, it is possible to objectively understand the gaze relationship between the participants in the conference, and it is possible to support the communication of the people participating in the conference between multiple points more easily than before. it can.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a multipoint video conference system according to a first embodiment of the present invention.

FIG. 2 is an example of a screen display in a conventional video conference system.

FIG. 3 is a configuration diagram of a conventional multipoint video conference system.

FIG. 4 is an explanatory diagram for explaining a conventional problem.

FIG. 5 is a diagram of a functional configuration example of a line-of-sight information transmission display unit according to an embodiment.

FIG. 6 is a diagram illustrating an example of multiplexing position / gaze partner data according to an embodiment.

FIG. 7 is an explanatory diagram of an example of calculation of a line-of-sight direction according to an embodiment.

FIG. 8 is a diagram illustrating an example of line-of-sight information display according to an embodiment.

FIG. 9 is a diagram illustrating a display example of a line-of-sight direction according to an embodiment.

FIG. 10 is a diagram illustrating a display example of a line-of-sight direction according to an embodiment.

FIG. 11 is a diagram illustrating an example of line-of-sight information display according to an embodiment.

FIG. 12 is a diagram showing an example of a screen display in the video conference system according to the embodiment.

FIG. 13 is a block diagram of a multipoint video conference system of a second embodiment.

FIG. 14 is a diagram showing an example of a screen display by the multipoint video conference system of the third embodiment.

[Explanation of symbols]

4 ... Decoder, 5 ... Image display unit, 6 ... Line-of-sight information transmission display unit, 7 ... Transmitting unit, 8 ... Receiving unit, 20 ... Multipoint video conference device, 30 ... Transmission line.

Claims (2)

[Claims] 1. An image of a conference participant of the own multimedia communication conference device is transmitted to another multimedia communication conference device, and the image of the conference participant is displayed on the own multimedia communication conference device as an image. In a multipoint multimedia communication conference system that performs multimedia communication, the conference participant of the own multimedia communication conference device is who of the other conference participants, or who of the conference participants of the other multimedia communication conference device. Is provided with gaze information generation means for generating gaze information indicating whether to gaze and transmitting the gaze information to another multimedia communication conference device, and gaze information display means for displaying the gaze information on the own multimedia communication conference device. Multipoint multimedia communication conferencing system characterized by. 2. An image of a conference participant of the own multimedia communication conference device is transmitted to another multimedia communication conference device, and the image of the conference participant is displayed on the own multimedia communication conference device as an image. In a multi-point multimedia communication conference system for performing multimedia communication, the self multimedia communication conferencing device receives gaze information indicating who to gaze from another multimedia communication conferencing device and displays the gaze information. An inter-multipoint multimedia communication conference system comprising gaze information receiving / displaying means.