JP2000175171A - Video image generator for video conference and video image generating method for video conference - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate an image of a virtual conference space that is natural and full of presence in a video conference system. SOLUTION: In the case of producing a video conference image that is going to express a conference form where conference participants make discussion around a table, the video conference is conducted among three points; a caption 21b indicates a portrait resulting from photographing a participant (b) at a point B, and a caption 22b depicts a portrait image resulting from extracting the person only from the image 21b and filling the other parts of the image 21b in black, a caption 21c is a portrait image resulting from photographing a participant (c) at a point C, and a caption 31 is an object image resulting from a conference desk being a foreground by means of a computer graphics. There are conference materials for the participants(b),(c) on the conference desk. A video conference image 41 that is exhibited for a participant (a) at a point A is obtained by superimposing the object video image 31 onto the other video images above on the basis of predetermined position and size information. In this case, the video conference image 41 to be exhibited for the participant (a) at the point A can be obtained by filling in advance unnecessary parts of the superimposed images in black.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video conference apparatus for generating a virtual conference space video having a sense of reality used in a video conference system and a method of generating the same.

[0002]

2. Description of the Related Art In a video conference system, a method of generating a virtual conference space image having a sense of reality is described in, for example, a literature (Journal of the Institute of Electrical Communication, Vol. 56, No. 598, 199).
6th, pp. 21-25, "Technical commentary by Mr. Nakatani and Mr. Arai," High-Presence Multimedia Communication Conference System "). In this document, as if the conference desk of the other party is connected to the conference desk of your own, the image of the "participant sitting in front of the desk" taken by the other party is projected on the large screen of your own Some are projected and displayed.

[0003] When a video conference is performed between three points using a conventional method, for example, Japanese Patent Laid-Open No. 6-113336, "3.
Dimensional multi-point video conference system "as described in the invention, in order to have a real and realistic video conference,
There is a screen in which a three-dimensional image is displayed on a split screen.

FIG. 4 shows an example of an image displayed on a screen. When the user's side is the point A, the opponent's side is the points B and C. A conference room image including the two participants and the conference desk photographed at the point B is projected and displayed on the screen 101 at the point A, and the screen C is displayed on the screen 102.
The conference room image including the three participants and the conference desk photographed in the above is projected and displayed.

[0005] Apart from the conventional method as described above, Japanese Unexamined Patent Publication No. 7-67035 discloses a "video conference video synthesizing system".
As described in, the images of multiple conference participants, whose background has been removed from the person video taken including the conference table and the conference materials on it, are combined and displayed on one background video There is also a way to do it.

FIG. 5 is a process diagram for synthesizing a conference space image using a conventional method. In a person image 201 photographed at a certain point, a background including a meeting desk and meeting materials is projected in addition to the person. Extract only people from this image,
A partially synthesized moving image 203 is generated by embedding a separately prepared background still image 202 around it. This generates a partial composite moving image 203 by fitting it onto the table image 204. By inserting this into the table image 204, a composite image 205 that is a final conference space image is generated.

[0007]

However, in the conventional video conference system, the video taken by the other party is projected on the screen as it is, and the conference room and the conference desk are fixed, so that conference participants can be seated. Locations are limited, especially when large numbers of people attend,
Layout was difficult, for example, the screen layout had to be changed.

In a conventional method of treating a conference desk or a person as a group of images and synthesizing it with another background image, when a conference space image is actually generated, a conference attached to a person image as shown in FIG. The video of the desk and the video of the conference desk in the new background are not smoothly connected, and the discontinuity 206
May cause an unnatural meeting space image.

Further, in the conventional video conference system,
By using the same conference desk at all points and using a shot of the conference desk as the background video, it is possible to generate a conference space video in which the person video and the background video are smoothly connected. Needs to be in a condition that the position of is fixed. Also, if the top of the conference table is not rectangular or if you have to take a picture of a person from an oblique direction,
It becomes difficult to smoothly combine a person image and a background image, particularly as if a conference desk were connected.

An object of the present invention is to provide a video conference image generation apparatus and a video conference video generation apparatus capable of generating a natural and realistic virtual conference space video in a video conference system. .

[0011]

According to the first aspect of the present invention, there is provided a video conference apparatus for generating a virtual conference space video for a video conference, wherein a background in the conference space is generated. A background image generation unit, a person image generation unit that generates a person image to appear in the conference space, an object image generation unit that generates an image of an object to be displayed in the conference space, and a person image and an object image A synthetic information storage unit for storing information on which position in the image is to be arranged, in which size, and in what order, and based on the information stored in the synthetic information storage unit, a person image and an object are placed on a background image. And a video synthesizing unit that generates a conference space video by superimposing the above video images.

In order to generate a virtual conference space video for a video conference, a background video generation unit, a person video generation unit, and an object video generation unit generate a background in the conference space and make a video of a person appearing in the conference space. Is generated, and an image of the object to be displayed in the conference space is generated. The synthesis information storage unit stores information on the position and size of the person image and the object image in the background image and in which order, and the image synthesis unit stores the information in the synthesis information storage unit. The conference space image is generated by superimposing the person image and the object image on the background image based on the obtained information. As a result, it is possible to generate a natural and realistic virtual conference space image.

According to a second aspect of the present invention, there is provided the video conference video generating apparatus according to the first aspect, wherein a video obtained by extracting only a person from a video of a video conference participant is prepared as a background video separately prepared. It is characterized in that a video image of a conference table is superimposed on a predetermined position at a predetermined size, and a video of the conference table is superimposed on the predetermined position at a predetermined size.

An image obtained by extracting only a person from an image of a video conference participant is superimposed on a predetermined position of a separately prepared background image at a predetermined size, and further, a video of a conference table is further superimposed thereon. Can be superimposed at a predetermined position in a predetermined size, so that a natural and realistic virtual conference space image can be generated.

According to a third aspect of the present invention, there is provided the video reproducing apparatus for a video conference according to the first aspect, wherein the image of the display board for display is displayed at a predetermined position of the separately prepared background image in the conference space. It is characterized in that a conference space image is generated by superimposing an image obtained by extracting only a part at a predetermined position at a predetermined size.

The image of the display board for display can be superimposed on the image obtained by extracting only the human portion at the predetermined position at the predetermined position at the predetermined position of the background image separately prepared in the conference space. , A meeting space image can be generated, and a virtual meeting space image with a sense of reality can be generated.

According to a fourth aspect of the present invention, there is provided a video conference image generation method for generating a video of a virtual conference space for a video conference, wherein a background in the conference space is generated and a person video to appear in the conference space is generated. Then, an image of the object to be displayed in the conference space is generated, and the person image and the image of the object are displayed at any position in the background image at any size.
In addition, information on the order of arrangement is stored, and a conference space image is generated by superimposing a person image and an object image on a background image based on the stored information.

[0018] Information on the position and size of the person image and the object image in the background image is stored, and based on the stored information, the person image is placed on the background image. Since the conference space video is generated by superimposing the video of the object and the object, it is possible to generate a natural and realistic virtual conference space video.

According to a fifth aspect of the present invention, there is provided the video conference image generating method according to the fourth aspect, wherein an image obtained by extracting only a person from a video image of a video conference participant is used as a background video image separately prepared. The present invention is characterized in that a first image is superimposed on a predetermined position at a predetermined size, and further a first image is superimposed on the predetermined position at a predetermined size.

[0020] An image obtained by extracting only a person from an image of a video conference participant can be superimposed on a predetermined position of a separately prepared background image in a predetermined size. By superimposing the first video at a predetermined position at a predetermined size, a virtual conference space can be generated as a continuous video, so that a sense of incongruity can be prevented.

According to a sixth aspect of the present invention, there is provided the video reproducing method for a video conference according to the fourth aspect, wherein the second video includes only a person portion at a predetermined position of the background video separately prepared in the conference space. A feature is to generate a conference space video by superimposing the extracted video at a predetermined size at a predetermined position.

In the conference space, the second video is superimposed on a predetermined position of a separately prepared background video at a predetermined position with a video obtained by extracting only the human portion at a predetermined position to obtain a conference space video. Can be generated,
It is possible to prevent the virtual conference space from being unnatural as a continuous image.

[0023]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described. In a video conference, a video conference video generating apparatus generates a video of a virtual conference space.

FIG. 1 is a block diagram illustrating the configuration of the embodiment of the present invention. In FIG. 1, a background video generation unit 1
Generates a background image in a conference space, a person image generation unit 2 generates a person image by removing a background portion from an image of a conference participant, and a object image generation unit 3 generates a person image. Generate an image of the object to be displayed in the conference space.

The composite information storage unit 4 stores information on the positions of the person image and the object image in the background image, and the image synthesizing unit 5 superimposes the background image, the person image, and the object image on the conference. Generate a spatial image.

When a virtual meeting space image with a sense of reality is generated, the constituent elements of the space are captured as two elements, a background and a person. In some cases, an unnatural image lacking in visual continuity is synthesized, for example, a noticeable boundary is formed between the conference room and the inside conference room. By the way, for example, when the conference desk is placed in front of the person and the background, the person, and the conference desk are displayed in this order from the back, it is considered that the natural display positions of the elements constituting the conference space are generally determined. Good.

The video elements constituting the conference space are considered as three types of background video, person video, and object video, and these videos are generated in parallel, and are determined according to the type of conference such as lectures and discussions. By superimposing and arranging the images in accordance with the order and position, a natural conference space image with a sense of reality is generated.

Next, the structure will be described in more detail with reference to FIG.
Generate a background video. The generated video can be realized, for example, by reading out a video file of a still image or a moving image stored in advance in a storage device such as a hard disk. In this case, the image may be an image of a virtual conference room created using computer graphics, as well as a live-action image of an existing conference room or the like. In addition to reading out a video file stored in advance, a video shot live may be used.

The person image generator 2 generates an image of a person participating in the conference. The image generated here is an image of only a person without a background. Such a video is generated, for example, by photographing a participant in front of a conference desk with a camera,
This can be realized by performing background removal processing on the video. To remove the background, that is, to extract only the person, cover the part to be removed with a blue cloth, etc., extract the part other than blue based on color information, specify using multiple cameras There is a method of extracting only a person based on the determined background and the distance to the person.

The image to be generated is not limited to a live-action person image, but may be a person image expressed by computer graphics. Japanese Patent Application Laid-Open No. 7-255044, "Animation type electronic conference room, video conference system and video conference method", describes in detail a method of visually expressing the shapes and actions of real conference participants as animated characters. I have. It is also possible to generate a virtual person image using such a method.

The background portion other than the person is painted in a certain color as a mask. By the way, in general, a person's image of a participant photographed at a remote place in a video conference is used. In this case, the captured video may be encoded, transmitted over a network, and combined at a point where video synthesis is performed. By providing such a mechanism in the person image generation unit 2, a multipoint connection video conference can be realized.

The object video generating section 3 generates a video of an object to be displayed as an object in the conference space separately from the background and the person. Video generation can be realized, for example, by reading a video file of a still image or a moving image stored in a storage device such as a hard disk in advance. In this case, the image may be not only a live-action video taken by photographing a real conference desk or a whiteboard, but also an image of a fictitious conference desk or whiteboard created using computer graphics.

The video file stored in advance may be used not only for reading out but also for live video. In addition, materials and 3
It is also possible to generate a final object video by expressing the dimensional objects in computer graphics and superimposing them on a conference desk or whiteboard represented by live action or computer graphics. If a background portion other than the object is filled with a specific color as a mask, an object image can be suitably generated.

The composite information storage unit 4 stores information on where to place a person image or an object image in a background image. Specifically, for a person image, for each of a plurality of person images, and for an object image, information on which position and size in the background image the object image is, for example, And stored in a table format.

At this time, information as to which coordinate position in the background image the upper left point of the rectangle including the video is to be matched as position information, and as the size information, the vertical and horizontal dots or pixels of the video are used as the size information. The information as to whether or not to arrange them can be used. In addition to such information on the position and size, information on whether to superimpose the images in the order of the background image, the person image, the object image, and the like is also stored.

When a certain person newly participates in the conference, it is necessary to synthesize a new image of the person, which has not been displayed so far, as a background image and display it. In addition, if the conference format changes from a situation where all participants discuss at the conference desk to a situation where one participant makes a presentation using the whiteboard, the video to be displayed, the position and size of the video, Everything in the matching order changes.
In consideration of various usage forms, information to be stored generally changes with time.

The video synthesizing section 5 generates a conference space video by synthesizing a background video, a person video, and an object video. Specifically, first, the background video generated by the background video generation unit 1 is fetched and stored in the video storage area. Next, an image to be superimposed on the background image is stored in the synthetic information storage unit 4.
Whether it is a person image or an object image,
Also, the captured image is superimposed on an appropriate position in the background image. In the captured video, a portion masked with a predetermined color is not replaced with a background video.

Next, the image that has not been synthesized is fetched from the person image generation unit 2 or the object image generation unit 3, and the background and the size information obtained by inquiring the synthesis information storage unit 4 first are used. By taking in pixel data of video and replacing it with pixel data of video,
The images are superimposed at appropriate positions in the background image. Also in this case, in the captured video, a portion masked with a predetermined color is not replaced with a background video.

FIG. 2 is a diagram showing an example of a video conference.
A description will be given of a procedure for generating a video conference video representing a conference in which a conference participant discusses by surrounding a table. It is assumed that a video conference is performed between three participants at a participant a at a point A, a participant b at a point B, and a participant c at a point C, and a video conference video to be presented to the participant a is generated. . It is easy to extend this to N points (N is a natural number).

Reference numeral 11 denotes a background image, and reference numeral 21b denotes a person image taken by the participant b at the point B, and a person image 22b in which only the person is extracted from this image and the other portions are filled in black. Reference numeral 21c denotes a person image of the participant c at the point C. Reference numeral 31 denotes an object video created by computer graphics of a conference desk serving as a foreground. On the conference desk, there are conference materials of the participants b and c, and these objects are also created by computer graphics.

By superimposing the object images 31 based on the information on the predetermined position and size,
A video conference video 41 to be presented ni to the participant a at the point A is obtained. At this time, the video conference video 41 to be presented to the participant a at the point A can be obtained by painting in black in advance. At this time, the area previously painted black is synthesized so that the lower image is displayed as it is.

FIG. 3 is a diagram showing another example of a video conference, in which a procedure for generating a video conference video representing a conference in which one of the conference participants gives a presentation using a large-displayed material is described. I do. It is assumed that a video conference is performed between three participants at a participant a at a point A, a participant b at a point B, and a participant c at a point C, and a video conference video to be presented to the participant a is generated. . This is called N point (N
Is easy to expand to a natural number).

Reference numeral 51 denotes a background image, and reference numeral 61b denotes a person image taken by the participant b at the point B, and a person image 62b in which only the person is extracted from this image and the other portions are painted black. Reference numeral 71 denotes an object image in which the meeting material and a device for displaying the meeting material are created by computer graphics. On the conference desk, there are conference materials of the participants b and c, and these objects are also created by computer graphics.

When creating the object video 71,
Portions other than the objects, in this case, portions other than the display device and the meeting materials, are painted black. As shown in the dotted frame (a), these images have an object image 71 on a background image 51 and a person image 6 on it.
By superimposing 2b on the basis of the information on the predetermined position and size, a video conference image 81 to be presented to the participant a at the point A is obtained. At this time, the area previously painted black is synthesized so that the lower image is displayed as it is.

Further, when presenting a video of a participant other than the person giving the presentation, a video of the participant c at the point C is presented.
By superimposing “c” on the object video 71 as it is, without particularly extracting the person region, a conference video 82 is obtained.

[0046]

According to the video conference video generating apparatus and method of the present invention described in detail above, a conference space video is generated by superimposing a person video and an object video on a background video. By doing so, it is possible to generate a natural and realistic virtual meeting space image.

Further, according to the video conference video generating apparatus and method of the present invention, it is extremely effective in realizing a virtual conference space video with a sense of reality in a video conference conducted between remote places. Therefore, the practical value can be increased.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an embodiment of the present invention.

FIG. 2 illustrates an example of a video conference.

FIG. 3 is a diagram showing another example of a video conference.

FIG. 4 is a conventional configuration diagram.

FIG. 5 is a process diagram for synthesizing a conference space image using a conventional method.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Background image generation part 2 Person image generation part 3 Object image generation part 4 Synthetic information storage part 5 Image synthesis part 11, 51 Background image

Claims (6)

[Claims] An image generation apparatus for a video conference that generates a virtual conference space video for a video conference, a background video generation unit that generates a background in the conference space, and a person video that appears in the conference space. A person image generation unit that generates an image of an object displayed in the conference space; an object image generation unit that generates the image of the person image and the object image; A combined information storage unit that stores information on whether to arrange in order, based on the information stored in the combined information storage unit, by superimposing the person image and the object image on the background image, A video synthesizing unit that generates the conference space video. 2. A video obtained by extracting only a person from a video of a participant of the video conference is superimposed in a predetermined size on a predetermined position of a separately prepared background video, and further, the conference 2. The video conference video generating apparatus according to claim 1, wherein the video of the desk is superimposed on a predetermined position at a predetermined size. 3. An image of a display board for display is superimposed on a predetermined position of a separately prepared background image in a predetermined size at a predetermined position in the conference space. 2. The video conference video generation apparatus according to claim 1, wherein the video of the conference space is generated. 4. A video conference image generating method for generating a virtual conference space video for a video conference, comprising: generating a background in the conference space; generating a person video to appear in the conference space; Generates an image of an object to be displayed therein, and stores information on the position and size of the person image and the object image in the background image and in which order, and the stored information. And generating a conference space image by superimposing the person image and the object image on the background image based on the background image. 5. A video obtained by extracting only a person from a video of a participant of the video conference is superimposed at a predetermined size on a predetermined position of the separately prepared background video. 5. The video generation method for a video conference according to claim 4, wherein the first video is superimposed on a predetermined position at a predetermined size. 6. In the conference space, a second video is superimposed on a predetermined position of a separately prepared background video at a predetermined size with a video obtained by extracting only a human part, at a predetermined size. 5. The method according to claim 4, wherein the conference space image is generated.