JP2000287188A - System and unit for inter-multi-point video audio communication - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow the title system to freely control the size of a portrait image of a communication opposite party and to enhance the transmission efficiency of the communication. SOLUTION: Each terminal displays a common three-dimensional space, enters position information to be located on the xy plane of the space, transmits a portrait and voice data of a use to a server together with the position information, the server recognizes positions of users 41 to 44 of the terminal 31 on the xy plane from the received position information, controls the resolution of video images from the terminals 31 to 33 in a way that the video image of the user 41 closer to the user 44 is displayed as a larger picture having a higher resolution and the video images of the users 42, 43 are displayed as a smaller picture having a smaller resolution in the case of viewing the three- dimensional space from the terminal 34 (user 44), and transmits the controlled result to the terminal 34. The server transmits video images from other terminals to each terminal while controlling the resolution.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for converting video information of a user imaged by a television camera of a terminal installed at a plurality of points and audio information input by a microphone,
The present invention relates to a multipoint video / audio communication system using a three-dimensional space that performs processing in accordance with a positional relationship in a three-dimensional space constructed in a network and outputs to a display device and a speaker of each terminal, and a device thereof. .

[0002]

2. Description of the Related Art Conventionally, a video conference system has been realized as a system for holding meetings between remote locations using audio information and video information. When a conference is held between a plurality of points using this system, a method of displaying on a plurality of display devices corresponding to the points, or dividing a screen of one display device and displaying a plurality of screens simultaneously there. Methods and the like are used.

[0003]

The above-mentioned prior art has a problem in that the number of connected points is limited.
There has been a problem that the size of the person image of the communication partner at another point cannot be freely controlled.
Also, regardless of the size of the displayed human image, the number of pixels of the video screen transmitted from another point is constant, that is, the amount of transmitted information is constant, and the transmission path cannot be used effectively. was there.

[0004] It is an object of the present invention to enable the size of a person image of a communication partner to be freely controlled, to approach a communication partner, to enable communication between users, and to make effective use of a transmission path. It is an object of the present invention to provide a multipoint video and audio communication system that can perform the above.

[0005]

In order to achieve the above object, in a multipoint video / audio communication system according to the present invention, a plurality of terminal devices and one server device are connected via a communication network, and each terminal device is connected. Means for capturing an image of a user, means for collecting a user's voice, means for inputting position information of the user in a three-dimensional space, and image information, sound information and position of the user Means for transmitting information to the server device, means for receiving position information, video information, and audio information of the user of the other terminal from the server device, and generating and receiving the image information in the three-dimensional space, the user of the other terminal Based on your location information,
Means for generating an object in the three-dimensional space of the generated image information and mapping the received video information of the other terminal user corresponding to the object, and image information of the three-dimensional space including the mapped video And means for reproducing the received audio information.

[0006] The server device includes means for receiving video information, audio information, and position information from each terminal; means for extracting a human image from the received video information; and a server based on the received position information of each terminal. Means for controlling the resolution of the extracted human image of each terminal user for each terminal; means for mixing the received audio information from each terminal under predetermined conditions; and Means for transmitting video information and position information of the controlled person image of each terminal user and the mixed audio information to the corresponding terminal.

[0007] In the invention constructed action as described above, the position information in three-dimensional space sent from the plurality of terminals a positional relationship between the end-to know the server device, the server device based on the positional relationship By changing the number of pixels of the video screen transmitted to each terminal, it is possible to transmit a screen with the number of pixels suitable for the size of the human image displayed on the terminal, and to efficiently use the transmission path Can be. Further, it is possible to select a partner with whom the user wants to have a conversation by operating the user and make a call between the users.

[0008]

An embodiment of the present invention will be described with reference to the drawings. Although this embodiment shows an example of four terminals, the present invention does not limit the number of terminals.
FIG. 1 shows the overall configuration of a multipoint video / audio communication system according to the present invention. The server device 1 is accommodated in a communication network 2 such as a LAN or a public telephone network, and terminals 31, 32, 33, and 34 installed at first, second, third, and fourth points are accommodated in the communication network 2. Have been. Each terminal 31, 32, 33, 3
4 generates a three-dimensional space, and transmits and receives video information, audio information, and position information to and from the server device 1 via the communication network 2, according to the position of the user of each terminal in the three-dimensional space. It displays a person image and outputs a conversation voice.

For example, as shown in FIG. 5A, the users 41 to 4 of the terminals 31 to 34 on an xy plane in a common three-dimensional space.
When the user 4 is located and the line of sight of the user 44 faces the wall surface 3, a three-dimensional space as shown in FIG. 5B is displayed on the display screen of the terminal 34 of the user 44, for example. The users 41 to 43 are displayed corresponding to the positions on the xy plane.

FIG. 2 shows information transmitted between the terminals 31 to 34 and the server device 1. The terminals 31 to 34 collect the position information of the terminal users 41 to 44 in the three-dimensional space (including the direction of the person), the person image of the user captured by the television camera, and the microphone. The sound of the user who made the sound is transmitted to the server device 1. On the other hand, the server device 1 manages all the positional information collected from each terminal, measures the positional relationship of the human image of each terminal based on these positional information, and
, The position information, the processed image, and the mixing sound of all the terminals except for the terminal are transmitted. For example, position information of the users 41 to 43, their images, and sounds are transmitted to the terminal 34.

FIG. 3 shows an embodiment of the server device 1 according to the present invention. Signals from the terminals 31, 32, 33, and 34 are transmitted to data demultiplexing units 11, 12, 13, 1 respectively.
4 and is separated into position information, video information, and audio information, respectively.
And the control unit 1 via the audio information bus 182, respectively.
5. Enter the video processing unit 16 and the audio processing unit 17. The control unit 15 manages the positional information of the user of each terminal in the three-dimensional space, measures the positional relationship of each user based on the positional information of each terminal, and controls the video processing unit 16 and the audio processing unit. 17
Output instructions to That is, with respect to the terminal 34, the distance from the position of the user 44 to each of the positions of the users 41, 42, and 43 is calculated, and according to these distances,
In the example of A, the person image of the short distance user 41 is large, and the person images of the long distance users 42 and 43 are small, and an instruction to display each is output. Is output, and users 42 and 4 at a long distance
An instruction not to output the sound of No. 3 is output. Such an instruction is output for each terminal. Control unit 1
At 5, the received position information of each terminal is transmitted to the control information bus 18.
3, all the data demultiplexing units 11, 12, 13,.
Distribute to 14.

The video processing section 16 receives the video information sent from each of the terminals 31 to 34, and first extracts only the human image portion by the human image extracting section 161. There are various methods for this extraction means. For example, if each terminal captures an image on a blue background, only a human image portion can be easily extracted by chroma key processing. The extracted person image is stored in the control unit 1
5, the resolution control unit 162 thins out each terminal to a resolution corresponding to the screen size of each person image displayed on the terminal. That is,
When the person is near in the three-dimensional space, the resolution is increased to enlarge the display screen of the person image. 3
If the person exists far away in the dimensional space, the resolution is lowered to reduce the display screen of the person image. For example, a screen with 100 × 100 pixels is changed to 50 × 50 or 2
A process is performed so that the screen has a number of pixels such as 5 × 25. In the processing for the terminal 34, in the example of FIG.
One person image is a large display screen with high resolution, and the person images of distant users 42 and 43 are low display screens with low resolution. In the process for the terminal 32, in the example of FIG. 5A, the person images of the user 42 of the terminal 32 to the user 43 of the terminal 33 are immediately close to each other and have a large display screen with the highest resolution. The distance from the user 42 is medium, the person image is a medium-sized display screen with a medium resolution, and the person image of the distant user 44 is a small display screen with a low resolution. In this way, the resolution of the person image of the user of the terminal other than the terminal is controlled for each terminal. The screen whose resolution is controlled is output from the person image distribution unit 163 to the video information bus 181.

The voice processing unit 17 receives voice information sent from each of the terminals 31 to 34 and, based on an instruction from the control unit 15, controls the voice mixing unit 171 according to the distance between users of each terminal. And mix the input audio.
For example, the sound of a user who is within a predetermined distance is mixed, and the mixed sound is output from the sound distribution unit 172 to the sound information bus 182. Terminal 3
5A, the voice from the near user 41 is sent to the terminal 34 in the case of FIG. 5A, but the voices from the distant users 42 and 43 are not sent to the terminal 34. The user 43 in FIG. 5A
Has moved to the position indicated by the broken line (43 ').
From 4, the users 41 and 43 'are relatively close and equidistant.
Therefore, both voices of the users 41 and 43 are mixed and transmitted to the terminal 34. In this way, for each terminal, the voice from the user of another terminal to be sent to this terminal is controlled.

Each of the data demultiplexing units 11, 12, 13, 1
In step 4, the position information of all the terminals except the own terminal, the processed video information and the mixed audio information are multiplexed and transmitted to the corresponding terminal. FIG. 4 shows a terminal 3 according to the present invention.
1, 32, 33 and 34 show an embodiment. Server device 1
The input signal enters the data demultiplexing unit 51 and is separated into position information, video information, and audio information. The position information input unit 55 is for controlling which position in the common three-dimensional space is to be placed. For example, the position information input unit 55 inputs a position where the user is located as xy coordinates in an xy plane in the three-dimensional space. . The three-dimensional space generation unit 54 updates the display screen of the three-dimensional space according to an instruction from the position information input unit 55. The input from the position information input unit 55 inputs not only the xy coordinates but also the line of sight. In the example of FIG. 5A, the position and the line-of-sight direction of the user 44 are input at the terminal 34, and a three-dimensional space image of the wall surface 3 of the common three-dimensional space is displayed on the display screen of the terminal as shown in FIG. As a result, the display screen is updated by the three-dimensional space generation unit 54. For example, a plurality of main positions in the common three-dimensional space and the surface of the common three-dimensional space viewed from the line of sight are stored in advance, and the position information input unit 55 is obtained from the stored screen information.
Selection according to the input position information, and updates the three-dimensional space display screen.

Further, the three-dimensional space generation unit 54 uses the position information of the other terminal user of the communication partner separated by the data demultiplexing unit 51 to convert the object of another user in the three-dimensional space of the image information. Generate. For example, in the example of FIG. 5, the display screen of the common three-dimensional space is determined in the terminal 34 as shown in FIG. 5B. At this time, the position of the user 44 is a point at the center of the lowermost end (foreground) of the screen, This screen position is used as the position information of the user 44, and based on the position information, the users 41 to 43 are used.
Are determined on the screen corresponding to the respective pieces of position information, and a user object is generated at those positions.

The video mapping section 52 maps the other terminal's personal video received from the data demultiplexing section 51 to the generated other user's object. The state of the person in the three-dimensional space including the mapped video is displayed on the display device 53 in a perspective (perspective). The person image is sent from the server device 1 as a square image having the number of pixels corresponding to the designated resolution, and maps, for example, the center of the square image to the corresponding user object position in the three-dimensional space screen.

The position information in the three-dimensional space input from the position information input unit 55 is input to the data demultiplexing unit 51.
Is transmitted to the server device 1 via the. The own image captured by the television camera 56 is also used by the data demultiplexing unit 5.
1 to the server device 1. More microphone 5
The own sound picked up by 8 is transmitted to the server device 1, and the mixed sound received from the server device 1 and separated by the data demultiplexing unit 51 is reproduced by the speaker 57.

FIG. 5 is a diagram for explaining a method of controlling the display screen based on the position of the user of each terminal in the three-dimensional space. Here, the persons 41, 42, 43, and 44 are the users of the terminals 31, 32, 33, and 34, respectively. FIG.
A indicates a person position on an xy plane in a three-dimensional space. This positional relationship is managed by the control unit 15 of the server device 1. 5B and 5C show screen examples on the display device of the user 44 of the terminal 34. Here, the user 44 is 3
The case where the wall surface 1 in the dimensional space is viewed is shown.

In FIG. 5A, when the user 43 of the terminal 33 comes to the position 43 ', the screen on the display device 53 of the user 44 of the terminal 34 is changed from FIG. 5B to C.
In this case, since the display screen of the user 43 is small in the case of FIG.
In the case of FIG. 5C, since the display screen of the user 43 is large, the server apparatus 1 performs control to transmit a high-resolution screen from the server apparatus 1. In the server device 1, the speaker 57 of the terminal 34, in the case of FIG.
5 (C) is output, and in the case of FIG. 5C, the mixing sound of the person 41 (the user of the terminal 31) and the person 43 (the user of the terminal 33) is output. Do.

If the display screen of the person is very small,
That is, when the person exists very far in the three-dimensional space, the person can be displayed as a small icon (pattern) without transmitting the person image. In the above embodiment, the size of the display screen is controlled based on the distance from the position of the user of each terminal to the position of another user.For example, the partner user on the display screen is moved from the terminal to the cursor, for example. It is also possible to change the size of the display screen in the server device 1 based on the instruction information and display the same on the display device of the terminal. Also, when the user points the other user on the display screen from the terminal with, for example, a cursor, the terminal supplies sound to the speaker, and a voice communication path is set between the two, so that the two users can talk with each other. It is also possible to perform such control.

[0021]

As described above, according to the present invention, the server device grasps the positional relationship between the terminals based on the positional information in the three-dimensional space sent from the terminals installed at a plurality of points. Generates video information and audio information transmitted from the server device to each terminal based on the positional relationship, so it can be transmitted as a screen with the number of pixels suitable for the size where the video information from each terminal is displayed It is. It is also possible for the user to select a partner with whom the user wants to have a conversation and to have a call between them. Furthermore, since only the screen data having the resolution actually displayed on the display device needs to be transmitted, there is an advantage that the transmission path can be efficiently used.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an entire multipoint video / audio communication system according to the present invention.

FIG. 2 is a diagram showing information transmitted between terminals 31 to 34 and a server device 1;

FIG. 3 is a diagram showing a functional configuration of an embodiment of the server device 1 according to the present invention.

FIG. 4 is a diagram showing a functional configuration of an embodiment of terminals 31 to 34 according to the present invention.

FIG. 5 is an exemplary view for explaining a method of controlling a display screen based on a position of a user of each terminal in a three-dimensional space.

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Tadahiko Komatsu 3-19-2 Nishi-Shinjuku, Shinjuku-ku, Tokyo Inside Japan Telegraph and Telephone Corporation (72) Inventor Hiroshi Otani 3-192-1, Nishi-Shinjuku, Shinjuku-ku, Tokyo No. Nippon Telegraph and Telephone Corporation F term (reference) 5C064 AA02 AC01 AC02 AC06 AC09 AC12 AC13 AC16 AC22 AD06 5C082 AA01 AA21 AA27 AA31 BA27 BA41 BB01 CB01 DA87 MM02

Claims (3)

[Claims] A means for capturing an image of a user; a means for collecting voice of the user; a means for inputting positional information of the user in a three-dimensional space; Means for multiplexing audio information and position information and transmitting the information to the server device, means for receiving position information, video information and audio information of the user of the other terminal from the server device, and generating image information of the three-dimensional space, Means for generating an object in the three-dimensional space of the generated image information based on the received position information of the other terminal user, and mapping the received other terminal user's video information corresponding to the object; A terminal device having a means for displaying image information in a three-dimensional space including the mapped video, a means for reproducing the received audio information, and receiving video information, audio information and position information from each terminal Means for extracting a person image from the received video information, and means for controlling the resolution of the extracted person image of each terminal user for each terminal based on the received position information of each terminal. Means for mixing the received audio information from each terminal according to a predetermined condition for each terminal, and image information and position information of a person image of each terminal user whose resolution is controlled for each terminal, and A multipoint video and audio communication system, characterized in that a server device having means for transmitting the mixed audio information to a corresponding terminal is connected via a communication network. 2. A terminal device of a multipoint video / audio communication system in which a plurality of terminal devices arranged at multipoints and one server device are connected by a communication network, wherein: a means for imaging a user; Means for collecting a user's voice; means for inputting position information of the user in a three-dimensional space; means for multiplexing the image information, sound information, and position information of the user and transmitting the multiplexed position information to a server device Means for receiving position information, video information, and audio information of a user of the other terminal from the server device; generating image information of the three-dimensional space, based on the received position information of the other terminal user, Means for generating an object in the three-dimensional space of the generated image information and mapping the received video information of the other terminal user corresponding to the object, and image information of the three-dimensional space including the mapped video Terminal device characterized in that it comprises means for displaying, and means for reproducing the received audio information. 3. A server device of a multipoint video and audio communication system in which a plurality of terminals arranged at multipoints and one server device are connected by a communication network, wherein video and audio information from each terminal is provided. Means for receiving position information; means for extracting a person image from the received video information; and resolution of the extracted person image of each terminal user for each terminal based on the received position information of each terminal. Means for controlling, the means for mixing the received audio information from each terminal according to a predetermined condition for each terminal, image information of a person image of each terminal user whose resolution is controlled for each terminal And a means for transmitting the position information and the mixed audio information to the corresponding terminal.