JP3610423B2 - Video display system and method for improving its presence - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a video display system suitable for a conference by connecting remote locations via a communication line and a method for improving the presence of the video display system.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, an electronic conference system and a videophone system are known as a video display system in which remote persons send their own videos to a partner. In an electronic conference system or a videophone system, an image captured by a video camera is input to a personal computer and sent from the personal computer to a personal computer that is an update partner via a communication line such as a LAN (wide area network).
[0003]
The other person's personal computer also sends the other person's image captured by the video camera to his / her personal computer. Both personal computers display the images taken by their own video cameras, for example, on the left side of the display screen of the CRT display device, and on the right side of the display screen the images sent from the other party, giving the user a sense of realism. It is.
[0004]
[Problems to be solved by the invention]
Thus, in mutual communication (full duplex communication) between two personal computers, only two people can participate in the conference, and electronic conferences at three or more locations cannot be held. Also, the video sent from the other party is for communication. There is a time lag with the video taken by the self video camera, and the display becomes unnatural. Furthermore, in the past, one's own video and the other's video were displayed in separate squares, and their positional relationship and display size were different for each user. For this reason, both cannot intentionally share the same space (screen position).
[0005]
Therefore, the present invention is to provide a video display system and a method for improving the presence, which can give a sense of presence that two or more people are having a meeting at the same place.
[0006]
[Means for Solving the Problems]
In order to achieve such an object, the invention of claim 1 includes a plurality of video input means for inputting video at different locations, a plurality of video display means for displaying videos at different locations, and the plurality of video inputs. Image combining means for combining a plurality of sets of images input from the means into one image so as to be a mirror image, and transferring the images combined by the image combining means to the plurality of image display means, The display is performed by the plurality of video display means.
[0007]
According to a second aspect of the present invention, in the video display system according to the first aspect, the image synthesis means is provided in a server, the plurality of video input means and the plurality of video display means are provided in a plurality of clients, The server and the plurality of clients are connected via a communication network.
[0008]
According to a third aspect of the present invention, in the video display system according to the first aspect, the video input means is a video camera, and the video display means is a projector.
[0009]
According to a fourth aspect of the present invention, a plurality of video input devices that input video are installed at different locations, a plurality of video display devices that display video are installed at different locations, and a plurality of sets that are input from the plurality of video input devices. Are combined into one video by an image processing device so as to become a mirror image, and the video combined by the image processing device is transferred to the plurality of video display devices and displayed on the plurality of video display devices. It is characterized by.
[0010]
According to a fifth aspect of the present invention, in the video display system according to the fourth aspect of the present invention, the image processing device is provided in a server, and the plurality of video input devices and the plurality of video display devices are provided in a plurality of ways. It is provided in a client, and the server and the plurality of clients are connected via a communication network.
[0011]
According to a sixth aspect of the present invention, in the video display system realism enhancing method according to the fourth aspect, the video input device is a video camera, and the video composition device is a projector.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings.
[0013]
(First embodiment)
FIG. 1 shows a system configuration of a video communication system according to the first embodiment of the present invention. In FIG. 1, a server 100 and a plurality of clients 200 are connected to a communication line (network) such as a LAN (local area network). Each client 200 captures an image of a person who wants to participate in the conference (including voice input), leaves an image of the person from the captured image, removes the background, and transmits the image to the server 100. On the server 100 side, the person images sent from the plurality of clients 200 side and the background prepared on the server 100 side are combined, and the combined video is transmitted to each client 200. Each client side displays the received composite video.
[0014]
A system configuration of the server 100 and the client 200 is shown in FIG. Since general-purpose personal computers and workstations can be used for the server 100 and the client 200, the description of the system will be simplified. 2, the CPU 1 on the client side performs system control according to the OS (Operating System), and executes the processing program shown in FIG. 3 to perform video communication according to the present invention. The ROM 2 stores part of the OS and system fonts. The RAM 3 stores input / output data for the CPU 1, an execution program loaded from the hard disk storage device (HDD) 8, calculation data of the CPU 1, data input from the keyboard 6, and the like.
[0015]
The CRT 4 displays a composite video described later. An input / output interface (I / O) 5 is connected to a LAN (local area network) and receives video transmitted from the server 100.
[0016]
The keyboard 6 inputs character information and the like and operation instructions for the CPU 1. The mouse 7 designates a specific position on the display screen of the CRT 4 and inputs an operation command, characters, numerical values, and the like related to information displayed at the position. The HDD 8 stores the OS and video communication program. The I / O 9 is connected to the video camera 10 and inputs video (moving image + sound) captured by the video camera 10.
[0017]
The server 100 can have almost the same hardware configuration as the client 100 described above. However, the video camera 10 is not connected to the server 100, and the HDD 8 is different from the client 200 in that the video composition and communication program (FIG. 4) and a background still image to be synthesized are stored. The content of the video communication program executed on the client 200 side is shown in FIG. 3, and the content of the video composition and communication program video communication program executed on the server 100 side is shown in FIG. The system operation of FIG. 1 will be described with reference to the flowcharts of FIGS.
[0018]
When the CPU 1 on each client 200 side receives an instruction to start the video communication program from the keyboard 6 or the mouse 7, the CPU 1 loads it from the HDD 8 to the RAM 3 and starts execution. When the CPU 1 on the server 100 side receives an instruction to start the video composition and communication program, the CPU 3 loads the video composition and communication program into the RAM 3 and starts executing the program.
[0019]
The client 200 connects to the server 100 using a conventionally known communication protocol, for example, a communication protocol such as TCP / IP (step S10).
[0020]
The CPU 1 on the client side stores the moving image (reference numeral 501 in FIG. 5) and sound input from the video camera 10 in the RAM 3, and then deletes the background image (reference numeral 503 in FIG. 5) from this moving image (step S20). → S39). For this purpose, an image processing method as described in US Pat. No. 5,572,248 can be used. In this image processing, a background image (moving image or still image) in which no person exists is acquired in advance, and the background is removed by performing mask processing on the input video with a background image in which no person exists. Furthermore, a mirror image of the human part image is created by left-right reversal.
[0021]
The moving image 504 in the form of a mirror image by removing the background image in this way, that is, the moving image image composed of the image data for the person 502 and the image data (505 in FIG. 5) in which each pixel of the background portion has a numerical value of 0, for example. The data 504 and the audio data are compressed and transmitted to the server 100 via the I / O 5 of the client 200 (step S40).
[0022]
Next, the CPU 1 of the client 200 receives the video (see reference numeral 602 in FIG. 5) on which the human image is synthesized on the server 100 side via the I / O 5, performs data restoration, and displays it on the CRT 4.
[0023]
On the other hand, the CPU 1 on the server 100 side connects in response to a communication request on the client 200 side, and communicates with the plurality of clients 200 in time series by multitask processing (step S100). First, the CPU 1 on the server 100 side receives the compressed video transmitted from each client 200 and temporarily stores it in the RAM 3 in the server 100 (step S110).
[0024]
When receiving images from all the connected clients 200, the CPU 1 on the server 100 side performs data restoration of the received images, and provides a plurality of person images (mirror image form) and a background prepared in the HDD 8 in the server 100. An image (reference numeral 601 in FIG. 5) is synthesized (step S130). The synthesized video 602 is compressed by the CPU 1 and transmitted to all clients 200 (step S140). Thereafter, the above-described processing is repeated until a termination instruction is given (loop processing of steps S110 to S160).
[0025]
In this way, as shown in FIG. 5, each client 200 can display an image in which all of the clients 200 have gathered just by performing mutual video communication with the server 100. Therefore, the communication program on the client 200 side can be the same as the conventional one. Further, if the background prepared on the server 100 side is a scenery of one room, it is possible to give a sense of presence as if everyone is in one room.
[0026]
(Second Embodiment)
The first embodiment is an example in which a video is transferred via a communication network and a composite image is displayed on a display device such as a CRT. An example in which image synthesis is performed optically is shown in FIG. In FIG. 6, 51 and 63 are video cameras, and 52, 53, 61 and 62 are projectors. Reference numeral 70 denotes a signal processing device that amplifies and distributes the transferred video signal. The image of the person photographed by the video camera 51 is sent to the projectors 53 and 61, projected by the projector 53 onto the screen 54 as a mirror image, and projected onto the screen 64 by the projector 61 as a mirror image. The video of the person photographed by the video camera 63 is sent to the projectors 52 and 61 and projected on the screens 54 and 64 so as to be mirror images. Accordingly, the images of the persons in the remote place are synthesized by the two projectors 52 and 53 (61 and 62) and displayed on the screen 54 (64).
[0027]
In the first and second embodiments described above, the conversation person number 11 is projected as a mirror image number 15 on the screen 54 and the conversation person number 12 is projected as a mirror image number 16 as shown in FIG. . A mirror image (number 17) of the person number 13 is also projected on the screen 54. Accordingly, the interrogators with the numbers 11 and 12 can see the mirror image 7 on the screen 54 and obtain a sense of presence as if the virtual person (18) is in their perception. The same can be said for the dialog number 13 in the remote location.
[0028]
Further, as shown in FIG. 8, when two persons (numbers 11 and 12) are in a remote place and are looking at the mirror images 13, 14, 15, and 16, the conversation person with the number 11 is the mirror image with the number 14. It is talking to the position of (the mirror image of the conversation person 2). The same is true for the number 12 dialogue person. This means that the talker 12 talks to the position of the virtual image 17 and both can talk. This principle holds even when there are three or more interlocutors.
[0029]
In addition to the above embodiment, the following example can be implemented.
[0030]
1) In the above-described embodiment, the server 100 side synthesizes the video, that is, the moving image and the sound. However, the composition target may be only the moving image.
[0031]
2) In the above-described embodiment, an example in which the video communication system is applied to an electronic conference has been described. However, the present invention can be applied to various types of information communication performed by a plurality of persons at remote locations.
[0032]
3) In the above embodiment, a plurality of clients are connected to the LAN. However, instead of the LAN, a communication network with a public telephone line interposed, for example, the Internet can be used.
[0033]
4) Priorities are set in the video camera, and when images of a plurality of persons are displayed in duplicate, display is performed in accordance with the above priorities, and overlapping portions are masked (removed) from images with lower priorities. It is also possible.
[0034]
5) There may be a plurality of persons at the same position on the display screen. Therefore, the distance between the video camera and the subject person is measured. If the position of a person is detected from the outline of the person image and the positions of the plurality of person images match, it is possible to display the person image with the smaller measurement distance with priority. Note that the distance (position) of the subject can be measured by installing a camera on the ceiling and measuring the position and distance of a person on the principle of a triangle, a position detection method using magnetism, and stereo stereoscopic viewing. It is preferable to use a known method such as a position detection method based on.
[0035]
6) In the first embodiment described above, the background removal processing and the conversion processing to the mirror image are performed on the client 200 side. However, when priority is given to the frame top feed speed, the video input from the video camera 10 is used. Is sent to the server 100 as it is, and image synthesis is performed on the server 100 side. At this time, the background removal process is not performed. Further, as the image processing on the server side, each person image can be compressed / enlarged so that the person images have substantially the same size.
[0036]
【The invention's effect】
As described above, according to the first and fourth aspects of the present invention, by combining and displaying a person image in a remote place into one in the form of a mirror image, the person who sees the display is as if he was a friend. You can feel like you are in the same room.
[0037]
In addition, a composite image can be created at the same scale, and the relative positional relationship between users becomes clear. Further, since the displayed video is a mirror image, non-linguistic dialogue such as pointing can be easily performed by using an analogy when looking at the mirror, and the sense of reality is further improved.
[0038]
According to the second and fifth aspects of the present invention, even when video is communicated via a communication network, no time lag occurs when each person image is displayed, and a high-quality display image can be obtained.
[0039]
In the inventions of claims 3 and 6, by optically synthesizing images with a projector or the like, a large number of persons can be displayed on a large screen, and a large number of persons can be handled.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a system configuration according to a first embodiment of the present invention.
2 is a block diagram showing a system configuration of a server 100 and a client 200. FIG.
FIG. 3 is a flowchart showing a processing procedure on the client side.
FIG. 4 is a flowchart showing a processing procedure on the server side.
FIG. 5 is an explanatory diagram for explaining video composition;
FIG. 6 is a configuration diagram showing a system configuration of a second embodiment.
FIG. 7 is an explanatory diagram showing a positional relationship between a person and a mirror image.
FIG. 8 is an explanatory diagram showing a positional relationship between a person and a mirror image.
[Explanation of symbols]
1 CPU
2 ROM
3 RAM
4 CRT
5,9 I / O
6 Keyboard 7 Mouse 8 HDD
10 Video camera

Claims (6)

A plurality of video input means for inputting videos captured at different locations including its own video, a plurality of video display means for displaying videos at different locations, and a plurality of sets input from the plurality of video input means Image synthesizing means for synthesizing the video into one video so as to become a mirror image, and transferring the video synthesized by the image synthesizing means to the plurality of video display means for display on the plurality of video display means A video display system characterized by: 2. The video display system according to claim 1, wherein the image synthesizing means is provided in a server, the plurality of video input means and the plurality of video display means are provided in a plurality of clients, the server, An image display system characterized by connecting clients via a communication network. 2. The video display system according to claim 1, wherein the video input means is a video camera, and the video display means is a projector. A plurality of video input devices that input different images including their own images are installed in different locations, a plurality of video display devices that display images are installed in different locations, and input from the plurality of video input devices A plurality of sets of videos are combined into one video by an image processing device so as to form a mirror image, and the videos synthesized by the image processing device are transferred to the plurality of video display devices and displayed on the plurality of video display devices. A method for improving the realistic sensation of a video display system. 5. The method for enhancing the realistic sensation of a video display system according to claim 4, wherein the image processing device is provided in a server, the plurality of video input devices and the plurality of video display devices are provided in a plurality of clients. And a method for improving the realism of a video display system, wherein the plurality of clients are connected via a communication network. 5. The method for improving the realistic sensation of a video display system according to claim 4, wherein the video input device is a video camera and the video processing device is a projector.

Images