JPH05183808A - Picture information input device and picture synthesizer device - Google Patents

Abstract

PURPOSE:To make it possible to synthesize a picture by rightly keeping a back and forth relation in a picture by making a photographing object into a model, arranging it in virtual spaces and operating the depth to the virtual object made into the model every picture element of photographing pictures of a camera based on the detection result of parameters such as the positions and directions, etc., of the camera at the time of photographing. CONSTITUTION:A scene modeling means 104 arranges a virtual object that a photographing object is made into a model by a shape modeling means 102 in virtual spaces. A camera 105 photographs a photographing object scene, and a camera parameter detection means 106 detects parameters such as the positions and directions, etc., of the camera 105 at the time of photographing and inputs the parameters in a picture element depth calculation means 107. Based on each parameter, the means 107 operates the depth to the virtual object when the virtual object is to be photographed by the camera 105 every picture element of a picture photographed by the camera 105. Thus, the picture of the means 102 can be synthesized by maintaining the back and forth realtion of an object in a picture kept rightly.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image information input device for calculating image information for creating a composite image and an image composition device for composing a plurality of images using the image information.

[0002]

2. Description of the Related Art Conventionally, as a technique for synthesizing a plurality of images input from a camera or the like, there is a method called chroma key. Chroma key is one of special effect techniques in broadcasting, and is a method for creating a composite image in which one image captured by two cameras is fitted with the other image. The principle of chroma key is described in, for example, “Broadcasting Technology Sosho Color Television” edited by Japan Broadcasting Corporation (196
1 year) pp. 539-540.

FIG. 5 shows an example of a conventional image synthesizing device using the principle of chroma key. In FIG. 5, two images 401 and 402 to be combined are the images 401 and 402.
The image 402 is input to the image synthesizing circuit 403 that synthesizes the image 402 into one image, and the synthesized image 404 is output.

The operation of the above image synthesizing circuit 403 will be described. Two input images 401 and 402 are input.
Of the signals of the above, a specific color or signal level in one of the images is used as a key, and when that key is detected, the signal of that part is replaced with the signal of the other image, and the images are combined and displayed. To do. For example, the car body 405 in the image 401.
When the portion (shaded portion) 406 other than the above has the same color or signal level and the images are combined using that as a key, the image combining circuit 403 inputs the pixels of the portion 406 of the image which is the key from the image 401. If that is the case, the pixel in that portion is replaced with the corresponding pixel in the other image 402 and output, and if the pixel in the portion 405 of the vehicle body is input, that pixel is output as is. By performing the above processing for one frame, the image 404 with the vehicle body 405 in front and the image 402 as the background is combined.

[0005]

However, in the conventional image synthesizing apparatus as described above, since the image to be synthesized has only the information on the color and the signal level and does not have the information on the depth, one image It was only possible to combine the images by always arranging in front of the other image or always arranging in the other image. For example, in FIG.
When synthesizing two images as shown in (a) and (b), synthesizing as shown in FIG. 4 (c) could not be performed. That is, the space between the front building and the back building in FIG.
It is not possible to synthesize an image in which the objects in the two images are intertwined with each other, such as the car passing through in (b), while maintaining the front-rear relationship of those objects, which is unnatural. There is a problem that only images can be obtained.

In consideration of the above-mentioned problems of the prior art, the present invention calculates an image information input device for outputting information related to depth necessary for performing composition while correctly storing the front-back relation of objects in an image, Another object of the present invention is to provide an image synthesizing device for synthesizing a plurality of images having depth information while properly maintaining the front-back relation of objects in the images.

[0007]

According to a first aspect of the present invention, a virtual object which is a model of a photographing object photographed by an image pickup device is arranged in a virtual space in substantially the same arrangement as the photographing object. Based on the parameters detected by the parameter modeling means for detecting parameters such as scene modeling means for making, the position, direction, angle of view, focal length and the like of the image pickup device at the time of photographing the object to be photographed,
And a depth calculation means for calculating the depth from the image pickup device to the virtual object when the virtual object placed in the virtual space is taken by the image pickup device, for each pixel of the image taken by the image pickup device. It is an image information input device.

According to the present invention of claim 2, the depth information of the pixels corresponding to a plurality of images is input, and the depth comparison means for comparing the depths of the depth information and the pixel information of the pixels of the plurality of images are input. An image synthesizing apparatus is provided with a pixel selecting unit that selects a pixel corresponding to the smallest depth according to the comparison result of the depth comparing unit and outputs pixel information of the selected pixel.

According to a fourth aspect of the present invention, the depth information of the pixels corresponding to a plurality of images is input, and the depth comparison means for comparing the depths of the depth information and the pixel information of the pixels of the plurality of images are input. Then, according to the comparison result of the depth comparing means, a pixel selecting means for selecting a pixel corresponding to the smallest depth and outputting pixel information of the selected pixel, and a depth of the pixel for which the smallest depth is selected. The image synthesizing device includes a depth selecting unit that outputs the information.

[0010]

According to the present invention of claim 1, the scene modeling means arranges the virtual object modeling the object to be photographed in the virtual space in substantially the same manner as the arrangement of the object to be photographed, and the parameter detecting means takes the image. A virtual object placed in a virtual space by detecting parameters such as the position, direction, angle of view, and focal length of the imaging device when the object is photographed, and the depth calculation means based on the parameters detected by the parameter detection means. The depth from the image capturing device to the virtual object when the image is captured by the image capturing device is calculated for each pixel of the image captured by the image capturing device.

According to the present invention of claim 2, the depth comparing means comprises:
The depths of the depth information of the corresponding pixels of the plurality of input images are compared, and the pixel selection unit is the smallest from the pixels of the pixel information of the plurality of input images according to the comparison result of the depth comparison unit. A pixel corresponding to the depth is selected, pixel information of the selected pixel is output, and a plurality of images are combined.

According to the present invention of claim 4, the depth comparing means comprises:
The depths of the depth information of the corresponding pixels of the plurality of input images are compared, and the pixel selection unit is the smallest from the pixels of the pixel information of the plurality of input images according to the comparison result of the depth comparison unit. The pixel corresponding to the depth is selected, the pixel information of the selected pixel is output, and the depth selection unit outputs the depth information of the selected pixel to synthesize a plurality of images.

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the drawings showing its embodiments.

FIG. 1 is a block diagram showing the arrangement of an image information input apparatus according to the first embodiment of the present invention.
That is, the information 101 on the shape of the object to be imaged (hereinafter, abbreviated as an object to be imaged) is input to the shape modeling means 102 that models the object to be imaged into a virtual object, and the output of the shape modeling means 102 is A scene modeling means 1 for arranging a virtual object in a virtual space together with information 103 on the arrangement of the object to be photographed in the scene.
Has been entered in 04.

On the other hand, from the camera 105, which is an image pickup device,
An image 110 of the photographed object is output, and parameters relating to the camera such as the position, direction, angle of view, and focal length of the camera 105 at that time are detected by the camera parameter detection means (parameter detection means) 106. The detection result is input to a pixel depth calculation unit (depth calculation unit) 107 that calculates the depth of each pixel of the captured image together with the output of the scene modeling unit 104. The pixel depth calculation means 107 outputs the image 108 of the virtual object generated by the calculation and the depth 109 for each pixel of the captured image.

Next, the operation of the above embodiment will be described.

First, the information 101 on the outer shape such as the size of the object to be photographed is measured in advance by means not shown and input to the shape modeling means 102. The shape modeling means 102 uses the input information about the shape 1
In order to model the object to be photographed into a virtual object using 01, the shape is approximated by (a set of) simple geometric shape primitives such as a polygonal plate and a sphere. Since the accuracy of the approximation affects the accuracy of the finally calculated depth,
Adjust the accuracy according to the usage of the depth value. The virtual object modeled by the shape modeling means 102 is
The scene modeling unit 104 uses the information 103 about the placement of each shooting target in the scene measured by a unit (not shown) to have the same positional relationship as the actual placement of the shooting target in the virtual three-dimensional space. Is arranged as.

On the other hand, the camera 105 photographs a scene to be photographed and outputs the photographed image 110. The camera parameter detection unit 106 detects parameters such as the position and direction of the camera 105, the focal length, and the angle of view when the image 105 is captured by the camera 105, and sends the parameters to the pixel depth calculation unit 107. The pixel depth calculation means 107 actually uses the input camera parameters to actually use the camera 10
An image 108 of a scene including a virtual object arranged in the virtual space is generated by the scene modeling unit 104 by calculation under the same conditions as when the image was captured in 5. In this case, the image 108 generated by calculation does not necessarily have to be output to the outside. Pixel depth calculation means 107
When the image 108 is generated, for each pixel of the image 108 corresponding to the pixel of the captured image, the depth to the virtual object closest to the camera seen in the direction of the pixel is calculated, and It is output as the depth 109 for each corresponding pixel of the captured image 110.

In this manner, the image 110 actually photographed by the camera 105 and the depth 109 to the object imaged in each pixel of the image 110 can be output.

Here, a method of generating the image 108 from the geometric shape primitives arranged in the virtual space by the scene modeling means 104 will be described. As this method, an image generation method in ordinary three-dimensional computer graphics can be applied. Three
FIG. 2 shows an example of the flow of image generation processing in three-dimensional computer graphics. First, the arrangement of the geometric shape primitives in the virtual space is expressed in an absolute coordinate space (called a world coordinate space) independent of the position of the camera 105. When the position and the direction of the camera 105 in the world coordinate space are determined, the visual field conversion processing 201 converts the position of the camera 105 into the coordinate space (this is the viewpoint coordinate space). Next, when the focal length and the angle of view of the camera are determined, the geometric transformation primitive 202 is transformed into the perspective coordinate space by the perspective transformation processing 202. In addition,
The visual field conversion process 201 and the perspective conversion process 202 are generally obtained by multiplication of a matrix and a vector, and are converted into two conversion processes 20.
It is also possible to concatenate the conversion matrices used in 1 and 202 and perform them collectively as one process. Next, the geometric shape primitive converted into the perspective coordinate space is projected on a screen for projecting an image. At this time, a plurality of geometric shape primitives may be projected on the same pixel on the screen. In such a case, the hidden surface removal processing 203 calculates the depths (distances) to all geometric shape primitives that are visible in the direction of the pixel on the screen, and compares them to be closest to the camera 105. The image obtained from the geometric primitive with depth is the image at that pixel. The concrete realization method of the conversion of these coordinates and the processing of hidden surface removal is
For example, it is described in some detail in "Computer Graphics," translated by Atsumi Imamiya, Japan Computer Association (1984).

The pixel depth calculating means 1 of the above embodiment
In 07, the case of using the method of calculating the depth through the view conversion, perspective conversion, and hidden surface removal processing has been described.
The depth calculation means is not limited to this method, and for example, 3
It is also possible to use analytical techniques such as "line-of-sight search method (or ray tracing method)", which is a known technique as an image generation method in three-dimensional computer graphics, without performing processing such as coordinate conversion.

FIG. 3A is a block diagram showing the configuration of the image synthesizing apparatus of the present invention according to claim 2, and FIG. 3B is a block diagram showing the configuration of the image synthesizing apparatus of the present invention according to claim 4. It is a figure. 3A and FIG. 3B have the same components with the same numbers. In this embodiment, an image synthesizing device for synthesizing three images is shown. That is, in FIG. 3A, the pixel 301 of the first image and the pixel 3 of the second image to be combined are displayed.
03 and the pixel information of each of the pixels 305 of the third image, the pixel selection unit 3 that selects the pixel closest to the camera.
09, the depth 302 corresponding to the pixel of the first image and the depth 30 corresponding to the pixel of the second image described above.
The depth information of each of the depths 306 corresponding to the pixels of the fourth image and the third image is input to the depth comparison unit 307 that compares the depths. The signal 308 of the comparison result of the depth comparing means 307 is the pixel selecting means 309.
The pixel information 310 of the pixel selected by the pixel selection unit 309 is output.

In FIG. 3B, the above-mentioned FIG.
And the depth 3 corresponding to the pixels of the first image
02, depth information 304 corresponding to the pixels of the second image, and depth information 306 corresponding to the pixels of the third image are input to the depth selection means 311 that selects the depth corresponding to the selected pixel. Then, the depth selection means 311 outputs the depth information 312.

Next, the operation of the above embodiment will be described.

Of the above embodiments, in the embodiment of the image synthesizing apparatus according to claim 2 shown in FIG.
Since the image synthesizing apparatus has the same configuration as that of the image synthesizing apparatus according to claim 3 shown in FIG. 3B except that there are no 11 and 312, the description of the operation will be given with reference to FIG. Only the described image synthesizing device is performed, and the description of the operation of the embodiment of the image synthesizing device according to claim 2 shown in FIG. 3A is omitted.

First, the pixels 30 of the first to third images
The pixel information of 1, 303, and 305 is input to the pixel selection unit 309 pixel by pixel for each corresponding pixel position of each image.
On the other hand, the depths 302, 304, 30 of the first to third images, which are the depths of those pixels 301, 303, 305.
The depth information of No. 6 is input to the depth comparison unit 307 and the depth selection unit 311. The depth comparing unit 307 compares the sizes of the depths 302, 304, and 306, determines the pixel of the image having the depth closest to the camera (usually, the smallest value when the depth is the distance from the camera), and the result is determined. Is sent to the pixel selection means 309 and the depth selection means 311. The pixel selection means 309 selects the pixel of the image determined to have the depth closest to the camera from the input pixels 301, 303, and 305 according to the signal 308, and outputs the pixel information 310 of the image as a synthesis result. To do. The depth selection means 311 uses the determination result signal 30.
The input depths 302, 304, 306 according to 8
From the pixel selection unit 309, and outputs the depth as depth information 312 of the pixel information of the image of the combined result output from the pixel selection unit 309.

In this way, the pixels and the depth of each pixel of an image obtained by combining a plurality of images while maintaining the front-back relationship of the objects in the images are obtained, and FIG.
A composite image as shown in FIG. 4C is obtained from the image shown in FIG.

In the above embodiment, the image synthesizing device for synthesizing three images is shown, but the number of images to be synthesized by the image synthesizing device of the present invention is not necessarily three, but two or more. Of course, it does not matter if you input any number of images.

Further, the pixel information and depth information of at least one of the plurality of images used for composition by the image composition apparatus of the above embodiment is the output information of the image information input apparatus according to claim 1. You may use.

Further, in the above embodiment, the depth selection means 3
Although the depth information of the selected pixel is output according to 11, the depth selection unit 311 is stopped and the pixel information 3 of the selected pixel is output, as shown in FIG.
It goes without saying that only 10 may be output.

Further, in the above embodiment, the images photographed by the cameras are combined, but the invention is not limited to this. If the image has depth information for each pixel, a virtual image such as computer graphics or animation is used. It may be an image such as. Alternatively, these images may be combined with the image captured by the camera.

[0032]

As is apparent from the above description, the present invention is provided with the depth calculating means for calculating the depth from the image pickup device to the virtual object for each pixel of the image photographed by the image pickup device. However, there is an advantage that the information related to the depth necessary for performing the composition can be calculated and output while correctly storing the context of the object in the image.

Depth comparison means for inputting depth information of corresponding pixels of a plurality of images and comparing depths of the depth information, and pixel information of pixels of a plurality of images for inputting depth information of the depth comparison means. According to the comparison result, the pixel corresponding to the smallest depth is selected, and the pixel selection means for outputting the pixel information of the selected pixel is provided. Therefore, a plurality of images having depth information are included in the image. There is an advantage that the objects can be combined while maintaining the proper context.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an image information input device of an embodiment according to the present invention of claim 1.

FIG. 2 is a flowchart showing an example of a processing flow in a depth calculation means that constitutes the image input apparatus according to the embodiment.

3A is a block diagram showing a configuration of an image synthesizing apparatus of the present invention according to claim 2, and FIG. 3B is a block diagram showing a configuration of an image synthesizing apparatus of the present invention according to claim 3; It is a figure.

4 (a), (b), and (c) are explanatory views of an image synthesizing example by the image synthesizing apparatus of the present invention.

FIG. 5 is an explanatory diagram of an example of image composition by a conventional image composition apparatus.

[Explanation of symbols]

 102 Shape Modeling Unit 104 Scene Modeling Unit 105 Camera 106 Camera Parameter Detecting Unit 107 Pixel Depth Calculating Unit 109 Depth 110 Captured Image 301 First Image Pixel 302 First Image Pixel Depth 303 Second Image Pixel 304 Pixel depth of second image 305 Pixel of third image 306 Pixel depth of third image 307 Depth comparing means 309 Pixel selecting means 310 Pixel information of synthetic image 311 Depth selecting means 312 Depth of pixel of synthetic image information

Claims (5)

[Claims] 1. A scene modeling means for arranging a virtual object, which is a model of an object to be imaged by an imaging device, in a virtual space in substantially the same arrangement as the object to be imaged, and the object to be imaged. Parameter detection means for detecting parameters such as position, direction, angle of view, and focal length of the imaging device at the time of shooting, and the virtual object arranged in the virtual space based on the parameters detected by the parameter detection means. When taking a picture with the imaging device, the depth from the imaging device to the virtual object,
An image information input device, comprising: a depth calculation means for calculating each pixel of an image captured by the image capturing device. 2. Depth comparison means for inputting depth information of corresponding pixels of a plurality of images and comparing depths of the depth information, and pixel information of the pixels of the plurality of images are input to obtain the depth. An image synthesizing device comprising: a pixel selecting unit that selects the pixel corresponding to the smallest depth according to the comparison result of the comparing unit and outputs pixel information of the selected pixel. 3. The image synthesizing apparatus according to claim 2, wherein depth information of pixels of at least one of the plurality of images is input from the image information input apparatus according to claim 1. 4. Depth comparison means for inputting depth information of corresponding pixels of a plurality of images and comparing depths of the depth information, and pixel information of the pixels of the plurality of images are input, and the depth is input. Pixel selection means for selecting the pixel corresponding to the smallest depth and outputting pixel information of the selected pixel according to the comparison result of the comparison means, and the smallest depth of the selected pixel for the selected pixel. An image synthesizing apparatus comprising: a depth selecting unit that outputs depth information. 5. The image synthesizing apparatus according to claim 4, wherein depth information of pixels of at least one of the plurality of images is input from the image information input apparatus according to claim 1.