JPH06165173A - System for attaining virtual society - Google Patents

Abstract

PURPOSE:To attain a virtual society as if many terminal users on remote places meet on the same place and make exchange. CONSTITUTION:Each of terminal equipments 1 to 3 is provided with a photographing device, an image receiver, a transceiver and a control information input device. A service center equipment 4 is provided with an image synthesizer, a sound mixer, an image distributer, a sound distributer, a background image source, a background sound source, and a control device. Each terminal equipment sends user's image, sound and control information to a communication network 5. The equipment 4 synthesizes the images and sounds of respective users with the background image and sound to creat a virtual society. The image and sound of the synthesized virtual society are sent to the terminal equipments 1 to 3 for respective user's through the network 5.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a virtual social networking system for providing a virtual exchange place to a plurality of people through a communication network by using image communication technology and voice communication technology.

[0002]

2. Description of the Related Art As a conventional similar system, a voice conference call is realized. Also, in so-called video conferences that use image communication technology, images are sent between multiple points, the images of the image receiver are divided, and multiple screens are displayed at the same time. The method of displaying is adopted.

[0003]

The above-mentioned prior art has a problem in that it is not possible to interact with each other at the same place as the other party who wants to interact with each other, and the number of points is limited in the communication between multiple points.

An object of the present invention is to allow a plurality of people who interact with each other to
Conversation, as if they were gathered in one place
It is an object of the present invention to provide a virtual social networking system that aims to eliminate the limit on the number of people who can interact with each other through gestures.

[0005]

A virtual social circle realization system of the present invention comprises a plurality of terminal devices and a service center device connected to them through a communication line, and each terminal device corresponds to the terminal user. An image pickup device for picking up an image of a person, a transmitter for inputting the voice of the user, an input device for inputting control information by the user, and a center for displaying a person video signal, a voice signal and control information of the user. A transmitting device for transmitting to the device, a receiving device for receiving the composite video signal and the mixed audio signal transmitted by the center device, an image receiving device for outputting the received composite video signal, and a transmitter for outputting the mixed audio signal The service center device includes a background video source for accumulating a plurality of background video signals, a background sound source for accumulating a plurality of background sound signals, a person video signal, an audio signal and a voice signal transmitted by each terminal device. A receiver for receiving control information,
A video synthesizing device for synthesizing a person video signal from each terminal device and a background video signal selected from the background video signal source, and a sound signal from each terminal device and a background sound signal selected from the background sound source are mixed. An audio mixing device, a distribution device that distributes the composite video signal and the mixed audio signal to each terminal device, and analysis of control information from each terminal device, and background video signals and backgrounds from the background video source and the background sound source. And a control device that controls selection of sound signals and control of image synthesis and voice mixing of the image synthesizer and the audio mixer.

[0006]

[Operation] The video and audio of the user in front of each terminal device are
It is converted into a video signal and an audio signal by the image pickup device and the transmitter, and sent out to the communication network. The video and audio signals of each user are sent to the service center device,
Therefore, it is combined with the background image and the background sound to create a virtual world "virtual social circle". The synthesized "virtual social circle" video and audio are sent to the terminal device of each user through the communication network and reproduced by the image receiving device and the handset.

[0007]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is an overall conceptual view of the virtual social circle realization system of the present invention, in which terminal devices 1, 2 and 3 are connected to a service center device 4 via a communication network 5 to receive virtual social circle services. It is shown that. In this embodiment, an example of three terminals is shown, but the present invention does not limit the number of terminals.

FIG. 2 shows an example of the configuration of each of the terminal devices 1, 2, and 3. The image pickup device 11 such as a television camera, the transmitter 12 such as a microphone, the image receiving device 13 such as a display, the receiver 14 such as a speaker, and a keyboard. And a control information input device 15 such as a mouse, and a transmission / reception device 16 including an encoding / decoding unit 161.

Here, the control information input device 15 is used to input the following control information. (1) Used to select “virtual social circle”. As a result, the user can select his / her desired “social circle” from among a plurality of “virtual social circles” prepared in the service center device. (2) The user image is used to select a position in the "virtual social circle", that is, in the space created by the background image and the background sound. This allows users to move around in a "virtual social circle." (3) When the user image is sent back to the user, it is used to select whether the image is the original image or the silhouette image. (4) It is used as an instruction to enlarge (up) the image of a specific opponent, selectively increase only the voice of the opponent, and reduce other voices. This allows a conversation with a particular person.

FIG. 3 shows an example of the configuration of the service center device 4, which is a code / code for a plurality of channels (three channels in this embodiment).
Transmitting / receiving device 4 including decoding units 411, 412, 413
1, a video synthesizing device 42 for synthesizing a plurality of video signals, a video distributing device 43 for distributing the composite video signals to a plurality of channels,
An audio mixer 44 for mixing a plurality of audio signals, an audio distributor 45 for distributing the mixed audio signals to a plurality of channels, a background video source 46 for storing a plurality of background video signals, and a background sound source 47 for storing a plurality of background audio signals. A position information storage device 48 that stores the position information of each person for each background, and a control device 49 that analyzes the control information from the terminal device side and controls each device. In addition, 420 is a video signal bus, 44
Reference numeral 0 is an audio signal bus and 490 is a control signal bus.

First, the basic operation of this system will be described. Now, it is assumed that users A, B, and C are using the terminal devices 1, 2, and 3, respectively. In addition, each user
An image shall be taken with a solid plain wall as the background. "Blue" is desirable as the color of the background wall. This is for facilitating video composition by chroma key processing described later.

Each terminal device 1, 2, 3 is connected to each user A,
B and C video and audio (collected as user image)
Are converted into video signals and audio signals by the image pickup device 11 and the transmitter 12, respectively, and "virtual social circle" selection information specified by the user is input as control information from the control information input device 15. The video signal, the audio signal, and the control information are transmitted from the transmission / reception device 16 to the service center device 4 through the communication network 5. The transmitter / receiver 16 includes a coding / decoding unit 1
61 is included and compresses video signals, audio signals, and control information for efficient transmission.

In the service center device 4, the transmission / reception device 41 receives the video signal, the audio signal, and the control information transmitted from the respective terminal devices 1, 2, and 3, and the respective encoding / decoding units 4 are received.
11, 412, and 413 are decoded and separated into a video signal, an audio signal, and control information.
It is sent to the video synthesizer 42, the audio mixer 44, and the controller 49 through the audio signal bus 440 and the control signal bus 460. The control device 49 analyzes the control information and selects a background video signal and a background sound signal corresponding to the “virtual social circle” designated by the user from the background video source 46 and the background sound source 47, respectively, and the video synthesizing device 42. To the voice mixing device 44. Further, the control device 49 refers to the position information memory 48,
The position information of each user A, B, C corresponding to the virtual social circle is sent to the video synthesizing device 42. The position information memory 4
A value designated by the user or an arbitrary value is stored in advance in 8. The video synthesizing device 42 synthesizes the video signal of each user and the background video signal according to the position information of each user and sends the synthesized signal to the video distributing device 43. Voice mixing device 4
4 mixes the voice signal of each user and the background sound signal and sends them to the voice distribution device 45. Video distributor 43 and audio distributor 4
5 distributes the combined image signal and mixed audio signal to the coding / decoding units 411, 412, and 413, respectively. Encoding / decoding sections 411, 412, and 413 encode the composite image signal and the mixed audio signal, respectively, and transmit them to the terminal devices 1, 2, and 3.

In each of the terminal devices 1, 2, and 3, the transmitting / receiving device 16 receives the composite image signal and the mixed audio signal,
The decoding unit 161 decodes and separates the composite video signal and the mixed audio signal. The image receiving device 13 displays a composite image, and the handset 14 outputs mixed voice. As a result, the users A, B, C in front of the terminals 1, 2, 3 can talk and interact as if they met at the same place.

Next, various methods of realizing image synthesis will be described. Now, it is assumed that each of the users A and BC selects positions such that the users are arranged in the order of A, B, and C from the left toward the background image. In this case, according to the instruction from the control device 49, the video synthesizing device 42, as shown in (1) of FIG.
The video signal of each user and the background video are combined by chroma key processing. In this state, when each of the users A, B, and C selects the silhouette image as the display method of their own image by the control information input device 15, this control information is transmitted to the control device 4.
9 and is a command to the video synthesizer 42. When synthesizing the video of each user and the background video, the video synthesizing device 42 converts the video of the corresponding user into a silhouette video. For the conversion process to the silhouette image, a method such as setting the black level in the portion extracted by the chroma key, or a method using a separately prepared silhouette image source fixed pattern can be considered. 4 (2), (3), and (4) show synthetic images corresponding to the terminals 1, 2, and 3 in the case of the fixed pattern.

In addition, a user may enter the control information input device 15
When the user instructs to move his / her position in the background image (for example, the moving direction, distance, etc. are instructed with a mouse or the like), this is transmitted to the control device 49, and the control device 4
9 updates the position information of the user in the position information memory 48, and uses it as a position command to the video synthesizing device 42. According to this position command, the video synthesizing device 42 moves the video signal of the corresponding user to the designated position and synthesizes it with the video signals of other users by chroma key processing. Further, at this time, the control device 42 refers to the position information memory 48 to calculate the relative positional relationship between the user who is instructed to move and the other user, and displays the image of the approaching user and the expansion of the visual field. If a command is given to the synthesizing device 42, it is also possible to follow the movement of the image of a certain user and enlarge (up image) the image / field of view of a specific partner for synthesizing.

Next, various methods for realizing voice mixing will be described. Now, as shown in FIG. 4, when the users A, B, and C are arranged in the order of A, B, and C from the left, the audio mixing device 44 adjusts the voice sound level of each user as in this arrangement. To do. In this case, if the audio channel is monaural,
You can adjust only the magnitude relationship, but in the case of stereo, adjust the audio that is distributed to the left and right. Also, the voice of a certain user should not be returned to that user. Further, when the image / field of view of a particular partner is expanded, only the voice signal of the partner is selectively increased and other voice signals are decreased. These controls are performed in synchronization with video composition by a command from the control device 49.

[0019]

According to the virtual social networking system of the present invention, a large number of users in remote areas can talk and interact in a natural manner as if they met at the same place. As described above, the system of the present invention provides a new system form for interaction between people.

[Brief description of drawings]

FIG. 1 is an overall configuration diagram of a virtual social circle realization system of the present invention.

FIG. 2 is a diagram showing a configuration example of each terminal device of FIG.

FIG. 3 is a diagram showing a configuration example of a service center device of FIG.

FIG. 4 is a diagram showing an example of a composite video displayed on each terminal device.

[Explanation of symbols]

 1 to 3 terminal device 4 service center device 5 communication network 11 imager 12 transmitter 13 image receiver 14 receiver 15 control information input device 16 transmitter / receiver 41 transmitter / receiver 42 image synthesizer 43 image distributor 44 audio mixer 45 audio Distributor 46 Background image source 47 Background sound source 48 Position information memory 49 Control device

Claims (5)

[Claims] 1. A plurality of terminal devices and a service center device connected to the terminal devices through a communication line, each terminal device capturing an image of a person of the terminal user, and a voice of the user. A transmitter for inputting, a input device for inputting control information by the user, a transmitting device for transmitting the person's video signal, audio signal and control information of the user to the center device, and the center device for transmitting The service center device has a receiver for receiving the composite video signal and the mixed audio signal, an image receiving device for outputting the received composite video signal, and a transmitter for outputting the mixed audio signal. A background video source for accumulating signals, a background sound source for accumulating a plurality of background sound signals, a receiving device for receiving a person video signal, an audio signal and control information transmitted by each terminal device, and each of the terminal devices. An image synthesizing device for synthesizing a person image signal of the person and a background image signal selected from the background image source; an audio mixing device for mixing an audio signal from each of the terminal devices and a background sound signal selected from the background sound source; A distribution device that distributes the composite video signal and the mixed audio signal to each terminal device, and analyzes the control information from each terminal device, and controls the selection of the background video signal and the background sound signal from the background video source and the background sound source, A virtual social circle realization system, comprising: a control device that controls video synthesis and voice mixing of the video synthesizer and the voice mixer. 2. When the control device receives a silhouette image instruction as control information from a certain terminal device, the person silhouette image is transmitted as a composite image signal for the terminal device instead of the person image signal transmitted by the terminal device. The virtual social circle realization system according to claim 1, wherein the video compositing device is instructed to transmit a signal. 3. When the control device receives an instruction to move the person image as control information from a certain terminal device, the control device causes the image synthesizing device to move the position of the person image of the terminal device in the background image. The virtual social circle realization system according to claim 1 or 2, wherein a command is issued. 4. The control device instructs the voice mixing device to adjust the volume of a voice signal other than the terminal device according to the position of the person image of the terminal device. Virtual social circle realization system described. 5. The control device enlarges the person image of a specific terminal device, raises the audio level, and reduces the person image from other terminal devices in accordance with the movement of the person image in the background image. 4. The virtual social circle realization system according to claim 3, wherein the video compositing device and the audio mixing device are instructed to lower the audio level. 5. The control device, in accordance with control information from a certain terminal device, designates and selects a visual field from the terminal device, enlarges a video image within the visual field, raises an audio level, and outputs an image outside the visual field. 3. The virtual social circle realization system according to claim 1, wherein the video compositing device and the audio mixing device are instructed to reduce the audio level and reduce the audio level.