JP2003219271A - System for synthesizing multipoint virtual studio - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a multipoint virtual studio synthesizing system for synthesizing videos from a plurality of cameras in real time without causing an uncomfortable feeling based on videos and depth information obtained by using the plurality of cameras. <P>SOLUTION: This multipoint virtual studio synthesizing system is provided with robot cameras 131 to 133 arranged at a plurality of places and for acquiring image data of objects and depth information of the image data, means 110 and 120 for controlling camera parameters including robot camera panning, tilting, zooming and focus in real time, means 141 to 143 for generating a key signal of each of the objects for synthesizing images on the basis of the depth information obtained in each of the robot cameras, an electronic plotting means 150 for generating electronic videos and depth information of the electronic videos to be used in image synthesis, and a means 160 for extracting image parts to be synthesized on the basis of the key signals of the objects and synthesizing the images. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for synthesizing video images of cameras at multiple points in video production.

[0002]

2. Description of the Related Art Conventionally, when synthesizing a person in a studio and an electronic image created using a computer,
Using a blue background (hereinafter referred to as a blue background),
Only the image of the person was cut out, judging that the part of the color other than blue was the image of the person. Then, the cut out image of the person is combined with the electronic image generated by the computer in association with the camera.

In addition, a post-production technique for synthesizing a plurality of images obtained by repeatedly photographing an object using a motion control camera may be used for synthesizing images. As described above, in the conventional technique, when the images are combined in real time, the images from one camera and the electronic background are combined, and when the real-time property is not required, the motion control
Background Art It has been performed that video images are repeatedly captured using a camera and then video images are combined.

[0004]

However, in the conventional method, there is a problem that images from a plurality of cameras cannot be combined in real time, and a method in which images are combined in real time using one camera is not included. However, there is a problem that an unnatural image discontinuity occurs around the composite part.

The present invention has been made to solve the above problems, and an object thereof is to obtain an image from a plurality of cameras in real time based on an image obtained by using a plurality of cameras and depth information. An object of the present invention is to provide a multi-point virtual studio compositing system for compositing images without any discomfort.

[0006]

In view of the above points, the invention according to claim 1 is arranged at a plurality of different points,
A robot camera for obtaining information including image data of a subject and depth information of the subject; means for controlling camera parameters including pan, tilt, zoom, or focus of the robot camera in real time; And a means for performing image composition by extracting an image part to be composed from the image data of the subject acquired by the robot camera based on the depth information.

With this configuration, a plurality of camera pans are controlled in real time and depth information is provided as a key signal, so that necessary portions of a plurality of images can be combined in real time based on depth. As a result, images from multiple cameras placed at various points can be displayed in different depths of each camera without creating a program by combining multiple images after complicated camera movements. By controlling based on the key signal, it is possible to realize a multipoint virtual studio composition system capable of unprecedented real-time image composition.

Further, the invention according to claim 2 is a robot camera arranged at each of a plurality of different points for acquiring information including image data of a subject and depth information of the subject, and a pan of the robot camera. , Tilt, zoom,
Alternatively, means for controlling camera parameters including focus in real time, means for generating a key signal of each subject to be image-synthesized based on depth information obtained for each robot camera, and based on the key signal And a unit for extracting an image portion to be combined from the image data of the subject acquired by the robot camera and performing image combination.

With this configuration, a plurality of camera pans are controlled in real time and the depth information is used as a key signal, so that necessary portions of a plurality of images can be combined in real time based on the depth. This allows images from multiple cameras placed at various points to be created based on each camera image and the key signal for the depth of that image, without the need to compose after performing multiple time-consuming camera movements. It is possible to realize a multi-point virtual studio compositing system capable of real-time image compositing that has never been achieved by controlling the above.

According to a third aspect of the present invention, in the first or second aspect, the multipoint virtual studio synthesizing system further generates an electronic image used in the image synthesizing and an electronic image for generating depth information of the electronic image. It has a structure provided with an image drawing means.

With this configuration, a plurality of camera pans are controlled in real time and depth information is provided as a key signal, so that necessary portions of a plurality of images can be combined in real time based on depth. As a result, until now, it is possible to combine one image and an electronic image with each other, or to combine multiple images placed at various points without the need to create the image after multiple rotations of the camera. By controlling the video of (1) based on each camera video and the key signal of the depth of the video, it is possible to realize a multipoint virtual studio compositing system capable of real-time image compositing that has never existed before.

[0012]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described in detail below with reference to the drawings. FIG. 1 shows the first of the present invention.
FIG. 3 is a diagram showing a block configuration of a multipoint virtual studio composition system according to the embodiment of FIG. In FIG. 1, the multipoint virtual studio composition system 100 includes a remote control platform 110, camera data processing means 120, a first robot camera 131, a second robot camera 132, a third robot camera 133, and a first image. Processing means 14
1, second image processing means 142, third image processing means 1
43, electronic drawing means 150, and image synthesizing means 160
Composed of and.

The remote control platform 110 and the camera data processing means 120 are communicably connected to each other, and
When remotely controlling each robot camera from the remote control platform 110 to capture an image using the three robot cameras 131 to 133, the remote control platform 110 is a camera that is required to capture a subject. Information on the operation (hereinafter referred to as camera operation information) is generated and the camera operation information is output to the camera data processing unit 120. In addition, the above connection that can communicate is
The connection may be wired or wireless, or may be via a network, and the same applies to communicable connections described below.

The camera data processing means 120 is communicably connected to the remote control platform 110 and the first to third robot cameras 131 to 133, and receives camera operation information output from the remote control platform 110 as an input. , The first to third robot cameras 131 to 1
Generate the camera data needed to control 33,
The camera data is transferred to the first to third robot cameras 131 to 13
3 and electronic drawing means 150 described later.

Therefore, the remote control platform 1
The first to third robot cameras 131 to 13 are remotely operated by the camera data processing means 120.
The camera data for 3 control is generated, and the first to third robot cameras 131 to 133 are the camera data processing means 12.
Shoot based on the camera data output from 0,
The remote control platform 110 is configured to be remotely controlled.

First to third robot cameras 131 to 133
Respectively correspond to the corresponding first to third image processing means 141 to 141.
143 is communicatively connected to the camera data processing unit 120. First to
The third robot camera 131 to 133 receives the camera data output from the camera data processing unit 120 as an input, performs an operation for shooting based on the input camera data, and captures a camera image and a subject obtained by the shooting. Data including depth information (hereinafter referred to as depth information) and the like are output to the corresponding first to third image processing units 141 to 143. Each of the first to third robot cameras 131 to 133 has, for example, Japanese Patent Application No. 2000-
The robot camera device disclosed in 383993 can be used.

Specifically, the first to third robot cameras 1
One or more robot cameras out of 31 to 133 are selected via the remote control platform 110, and the camera platform of the selected robot camera or the pan, tilt, zoom, focus, etc. is selected by the remote control platform 110.
The robot camera selected by remote control via the camera shoots an image.

First to third image processing means 141 to 143
Are the corresponding first to third robot cameras 131.
To 133, the camera image and depth information, etc., are input, and image data (corresponding images are referred to as reference symbols 31, 32, 33, respectively) for each corresponding camera image.
It is shown using. ), A key signal of the subject is generated based on the depth information, and the generated key signal is output to the image synthesizing means 160.

Here, the depth information for generating the key signal can be generated based on the depth information inclusion data by using, for example, the stereoscopic information detection method and apparatus disclosed in Japanese Patent Laid-Open No. 2000-121339. In addition, Park, Inoue, "Generation of arbitrary video using multiple cameras",
IEEE ICIP'97, Minutes, 1997, 149
−152 (Jong-Il Park and Seiki Inoue, “Arbitr
ary View Generation from Multiple Cameras ”, Proc,
It is possible to generate depth information using the 5-eye stereoscopic view disclosed in IEEE ICIP'97, 1997, pp149-152).

FIG. 2 shows each of the first to third image processing means 141.
14A to 14C are diagrams for explaining a method of generating a key signal. In FIG. 2, the second robot camera 13 is shown.
2 is an image 32 that is captured based on the depth information obtained by capturing the image 32.
2 and the key signal 32-3 to be generated.

Therefore, the first to third robot cameras 1
31 to 133 have the function of any one of the above special cameras. Here, for convenience of explanation, the brightness value (0-255) of the captured image signal is
It is assumed that each image corresponds to depth (Zfar to Znear). Here, assuming that the brightness in the area serving as the key signal of the subject is equal to or higher than a predetermined brightness value, a key signal as shown in image 32-3 can be generated.

It is necessary to select which subject is to be synthesized and generate a key signal of the subject before performing the image synthesis processing based on the information output from the image processing means. An example of the key signal generation process when the subject is a person will be described with reference to FIG. To generate the key signal, first, a known image processing technique is used to detect the skin color, and the image signal of the portion where the skin color is detected is extracted. Figure 2
Then, the image obtained by extracting the skin color based on the image data (see the image 32) obtained by the second robot camera 132 is indicated by a quotation symbol 32-4. here,
For the skin color detection and extraction, for example, a known image processing device (trade name, Quick Mag, Applied Measurement Technology Laboratory Co., Ltd.) or the like can be used.

Next, depth information about the range where the skin color is extracted is generated. An image 32-5 shows an image of a depth signal (hereinafter, referred to as a depth key of a skin color portion) corresponding to the skin color extracted range. Finally, for the key signal, for example, the average value K of the depths generated based on the depth key of the flesh-colored portion is given a width 2Δ, and an area having a depth of K ± Δ is designated based on the depth information. , Is generated by using the signal for the area as a key signal (see image 32-3).

As a method of determining the position of a person by a method other than skin color extraction, a method of attaching a three-dimensional position sensor to an object can be used instead of determining the position of a person based on image data. Is. Further, instead of generating the depth key signal, it is possible to specify the range of depth and extract the specified range directly from the image.

The electronic drawing means 150 receives the camera data output from the camera data processing means 120 as input, and creates an electronic image 51 and a depth key signal for the electronic image 51 based on the input camera data, It is output to the image synthesizing means 160.

The image synthesizing means 160 receives the image data, key signals, etc. from the first to third image processing means 141 to 143, the electronic image 51 from the electronic image forming means 150, and the key signal of the electronic image 51 as inputs. , Generates a composite image based on each key signal. In FIG. 1, an image taken by the third robot camera 133, an image of a person extracted from the images taken by the first robot camera 131 and the second robot camera 132, and the image synthesizing unit 15 are shown.
An example in which the electronic image 51 generated by 0 is combined is shown as an image 61.

In the image synthesizing process, in order to synchronize the data from the cameras at a plurality of points, a known frame delay device is used. By adjusting the delay amount of the image data to be processed, the reception timing of each image data can be synchronized.

As an example of the image thus synthesized,
In FIG. 3, images 31-3, 32-3, 33 of key signals generated by the first to third image processing means 141 to 143.
-3, the image 51-3 of the depth key signal generated by the electronic drawing means 150 is synthesized by the image synthesizing means 160, and the image of the synthesized key signal and the finally obtained synthesized image 61 are shown. The image of each key signal used here is accompanied by a depth signal, and the front-rear relationship of the images to be combined can be adjusted based on the information. For example, by stacking the images in the back one by one, the images in the foreground can be displayed in the foreground.

As described above, the multipoint virtual studio synthesizing system according to the first embodiment of the present invention controls a plurality of camera pans in real time and has depth information as a key signal, so that The necessary parts of the image can be combined in real time based on the depth. With this, until now, one video and electronic video can be combined,
Controlling images from multiple cameras placed at various points based on each camera image and the key signal of the depth of that image, without the need to spend time and effort to rotate the cameras multiple times and then synthesize them By doing so, real-time image composition has never been possible.

[0030]

As described above, according to the present invention, a multipoint virtual image is synthesized in real time from images from a plurality of cameras on the basis of images obtained by using a plurality of cameras and depth information. A studio composition system can be realized.

[Brief description of drawings]

FIG. 1 is a diagram showing a schematic configuration of a multipoint virtual studio composition system according to a first embodiment of the present invention.

FIG. 2 is a diagram for explaining a key signal generation method in each image processing means of the multipoint virtual studio composition system according to the first embodiment of the present invention.

FIG. 3 is a diagram for explaining an image synthesizing method in the image synthesizing means of the multipoint virtual studio synthesizing system according to the first embodiment of the present invention.

[Explanation of symbols]

31, 31-3, 32, 32-2, 32-3, 32--
4, 32-5, 33, 33-3, 51, 51-3, 6
1, 61-3 Image 100 Multipoint virtual studio composition system 110 Remote control platform 120 Camera data processing means 131, 132, 133 Robot cameras 141, 142, 143 Image processing means 150 Electronic drawing means 160 Image combining means

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Hideki Mitsumine             1-10-11 Kinuta, Setagaya-ku, Tokyo, Japan             Broadcasting Association Broadcast Technology Institute (72) Inventor Takashi Fukaya             1-10-11 Kinuta, Setagaya-ku, Tokyo, Japan             Broadcasting Association Broadcast Technology Institute F-term (reference) 5B057 BA02 CA01 CA08 CA12 CA16                       CB01 CB08 CB12 CB16 CC01                       CE08                 5C022 AB61 AB62 AB65 AB66 AC01                       AC69                 5C023 AA11 AA37 BA01 BA09 BA12                       CA03 DA04 DA08

Claims (3)

[Claims] 1. A robot camera arranged at each of a plurality of different points for acquiring information including image data of a subject and depth information of the subject, and a pan, tilt, zoom, or focus of the robot camera. A means for controlling the included camera parameter in real time, and a means for extracting an image portion to be combined from the image data of the subject acquired by the robot camera based on the depth information and performing image combination. A multipoint virtual studio synthesis system characterized by 2. A robot camera arranged at each of a plurality of different points for acquiring information including image data of a subject and depth information of the subject, and a pan, tilt, zoom, or focus of the robot camera. Means for controlling in real time camera parameters including: means for generating a key signal of each subject to be image-synthesized based on depth information obtained for each robot camera; and the robot based on the key signal A multipoint virtual studio synthesizing system, comprising: means for extracting an image portion to be synthesized from image data of a subject acquired by a camera and performing image synthesis. 3. The multipoint virtual studio composition system,
3. The multipoint virtual studio composition system according to claim 1, further comprising an electronic drawing means for generating an electronic image used in the image composition and depth information of the electronic image.