JPH0630336A - Picture input device and picture synthesis method - Google Patents

Abstract

PURPOSE:To provide the picture synthesis method and the picture input device to obtain an input picture in which a problem of picture synthesis in a conventional method that one picture is always arranged before or after the other picture is solved and the pictures are synthesized while keeping the preceding and succeeding relation of objects in the pictures. CONSTITUTION:A picture processing section 104 extracts a picture element area corresponding to an object from a picture inputted from an image pickup section 102, a distance measurement section 106 measures a distance up to a point on the object, a distance calculation section 108 calculates the distance of a picture element in a picture element corresponding to the object from measured distance data through interpolation or extrapolation and adds the result to picture element data 103 as depth data 109 and the added data are outputted. Thus, the picture synthesis while comparing the depth with other depth data is implemented and the picture input for the purpose is attained.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for synthesizing a plurality of images and an image input device for obtaining the input images.

[0002]

2. Description of the Related Art As an example of a conventional image synthesizing method, a method called chroma key which is widely used in the field of broadcasting will be described. For the principle of chroma key,
For example, "Broadcasting Technology Shusho Color Television" edited by Japan Broadcasting Corporation (1961) pp. 539-540. Chroma key is one of the special effect technologies in broadcasting, and is basically a method for creating a composite image in which one of two images taken by an image input device such as a camera is fitted with the other image. .

FIG. 6 shows an example of a conventional image synthesizing method using the principle of chroma key. In FIG. 6, 601 and 60
Reference numeral 2 is two images to be combined, 603 is an image combining device, and 604 is a combined image. Image synthesizer 60
Reference numeral 3 designates a specific color or signal level in one of the two input images 601 and 602 as a key, and when a key is detected, the signal of that portion is used for the other image. To generate a composite image.

Note that the principle of the chroma key is also applied when displaying an image captured from the outside on the display screen of a computer. That is, by specifying the area for displaying the captured image on the frame buffer that stores the image data on the computer side and filling the area in advance with the key color, the key A composite image is generated by replacing only the part with an image captured from the outside.

[0005]

However, in the conventional image synthesizing method, since the input image to be synthesized does not have depth information, one image is always placed in front of the other image. I could only combine images that had the effect of appearing either, or always behind. For example, when synthesizing the two images shown in FIGS.
As shown in FIG. 7 (c), the car in FIG. 7 (b) passes between the front building and the back building in (a).
The images in which the objects in one image are intricately in front of and behind each other cannot be combined while maintaining the front-to-back relationship of those objects, and only an unnatural image is obtained.

In view of the above point, the present invention has an object to provide an image input device and an image synthesizing method for synthesizing while maintaining the front-to-back relationship between individual objects in an input image.

[0007]

In order to solve the above-mentioned problems, an image input apparatus of the present invention comprises an image pickup section for picking up an image and outputting pixel data of the image, and pixel data of the image outputted by the image pickup section. An image processing unit for extracting pixel regions of one or more subjects from the image capturing unit; and one or more on the subject designated by the image capturing unit for each one or more subjects to be extracted by the image processing unit. A distance measuring unit that measures the distance along the optical axis direction of the imaging unit up to three points, and the distances to the pixels corresponding to the measured one to three points as necessary. By interpolating or extrapolating, the depth of each pixel is calculated, and the calculated distance value is added to the pixel data of each subject output by the image processing unit and output, and with respect to the pixel data of other regions. The depth of a predetermined value It is configured to and a distance calculation unit which imparts to the output as data.

In the image synthesizing method of the present invention, the pixel data and the depth data of the image output by the image input device of the present invention are used as at least one input, and one or a plurality of pixel data and depth data for each pixel are provided. The image is taken as another input, and the depth data is compared for each pixel at the corresponding pixel positions of all the input images, and the pixel that is closest, that is, the pixel with the smallest depth data, is the pixel at that pixel position in the image after synthesis. Is output as.

[0009]

According to the present invention, according to the above configuration, the image pickup unit picks up an image and outputs pixel data of the image, and the image processing unit outputs one or more objects from the pixel data of the image output by the image pickup unit. Of the pixel area of the
Target of extraction from the image capturing unit by the image processing unit 1
The distance along the optical axis direction of the image pickup unit to one or three points on the subject designated for each one or a plurality of subjects is measured for each subject, and the distance is measured according to the number of points to be measured. The distance calculation unit can calculate the distance by interpolation or extrapolation, add depth data to the pixel data of each subject output by the image processing unit, and output the pixel data.

Further, the pixel data and the depth data of the image output by the image input apparatus of the present invention are used as at least one input, and the pixel data and the depth data for each pixel are stored in the image data.
One or more images are taken as another input, and the depth data is compared for each pixel at the corresponding pixel positions of all the input images, and the closest pixel, that is, the pixel having the smallest depth data is By outputting as the pixel at the pixel position, it is possible to generate a composite image in which the front-back relationships between the individual objects in the plurality of images are correctly preserved.

[0011]

Embodiments of the present invention will be described in detail below with reference to the drawings.

(Embodiment 1) FIG. 1 shows the present invention (claims 1 to 1).
It is a block diagram which shows one Example of the image input device of 3). In FIG. 1, 101 is one or more subjects to be image input targets, 102 is an image capturing unit configured by a camera, 103 is pixel data of an image captured by the image capturing unit 102, and 104 is pixel data 103. An image processing unit that extracts a pixel area corresponding to one or more instructed subjects 101, 105 determines which pixel is a pixel of which subject 101, or which pixel is not a subject. The signal 106 indicates a subject 1 for each one or a plurality of subjects 101 to be extracted.
01 is a distance measuring unit that measures the distance to one or three points, 107 is distance data measured by the distance measuring unit 106, 108 is depth data corresponding to each subject according to the signal 105, and pixel data is calculated. A distance calculation unit that is output in association with 103 and 109 is depth data that is output by the distance calculation unit 108.

2 and 3 are diagrams for explaining the operation of the image input apparatus of this embodiment shown in FIG.
3 is a layout drawing showing an example of an image input device of this embodiment and an object at the time of image input, and FIG. 3 is an image input from the image input device shown in FIG. 2 and 3 will also be used in the description of another embodiment described later.

In FIG. 2, 201 is a camera that constitutes the image pickup unit 102, 202 is a rangefinder that constitutes the distance measurement unit 106, 203 is a rectangular parallelepiped which is one example of the subject to be extracted, and 204 is an extracted object. A cylinder, 205, 206, 207, which is an example of another subject as a target, shows three points on the subject 203, and 208, 209, 210 show three points on the subject 204. The number of subjects is not limited to this.

For the following description, the optical axis direction of the camera 201 (that is, the direction perpendicular to the lens surface) is the Z axis, the direction perpendicular to the floor surface is the Y axis, and the direction parallel to the floor surface is the X axis. Camera 2
Assume a left-handed coordinate system whose origin is 01.

In FIG. 3, the shaded area 301 is the subject 20.
The horizontal line portion 302 represents the pixel area corresponding to 3
A pixel region corresponding to the pixel region 303, a pixel region corresponding to the point 205, a pixel region 304 corresponding to the point 206, a pixel region 305 corresponding to the point 207, a pixel region 306 corresponding to the point 208, and a pixel region 307. 209 indicates a pixel corresponding to 209, 308 indicates a pixel corresponding to the point 210, and 309 indicates a pixel region which is not included in either the shaded area 301 or the horizontal area 302.

The image processing unit 104 extracts various characteristic amounts (for example, color, shape, texture, etc.) from the image shown in FIG. 3 and compares them with the characteristics of the objects 203 and 204 to obtain each object. Pixel regions 301 and 302 corresponding to 203 and 204 are extracted, and for each pixel, it is determined which pixel 203 corresponds to which subject 203 or 204, or which pixel 309 does not correspond to any subject 203 or 204. Generate an identifying signal 105,
It is sent to the distance calculation unit 108.

The operation of three different modes in each of the following image input devices will be described. First, in the image input device in the first mode, the distance measuring unit 106 uses the distance meter 202 to set one point (here, the point 20 on the subject 203).
Take 5 as an example. ) And one point on the subject 204 (here, the point 208 is taken as an example), the distance along the Z-axis is measured. As a distance measuring method of the distance meter 202, a method using response time or phase difference of reflected pulse of ultrasonic wave or laser light can be used. The distance calculation unit 108 determines the distance to the point 205 according to the distance data 107 input from the distance measurement unit 106 and the color identification signal 105 input from the image processing unit 104, and the depths of all pixels included in the pixel region 301. As the data 109, the distance to the point 208 is output as the depth data 109 of all the pixels included in the pixel area 302, and the pixels 309 of the areas not included in any of the pixel areas 301 and 302 are determined in advance. A constant value (usually the maximum value indicating infinity) is output as the depth data 109.

Next, in the image input device in the second mode, the distance measuring unit 106 uses the rangefinder 202 to detect two points on the object 203 (here, the points 205 and 206 are taken as an example) and the object. 2 points on 204 (here, point 208)
Take point 209 as an example. ) Is measured along the Z-axis. The distance calculation unit 108 follows each of the subjects 20 according to the distance data 107 input from the distance measurement unit 106.
The distance is calculated by the following interpolation / extrapolation calculation for each of 3 and 204. Since the calculation method is common to the subjects 203 and 204, only the subject 203 will be described.

First, the subject 203 is the floor surface (XZ plane).
Assuming a plane perpendicular to, the equation for this plane is given by AX + BZ + C = 0. Coordinates (X) obtained from the pixel position in the image of the pixel 303 corresponding to the point 205
1, Y1) and the distance to the point 205 (Z1) measured by the distance measuring unit 106, the coordinates (X2, Y2) obtained from the pixel position in the image of the pixel 304 corresponding to the point 206, and the distance measuring unit. The distance to the point 206 measured at 106 (Z
From 2), two points (X1, Y1, Z1) and (X2, Y2,
Assuming that Z2) is a point on this plane, the coefficients A, B, C of the equation of the plane are obtained.

Unless the X and Z coordinates of these two points are equal, the coefficients A, B and C are uniquely determined. If both X and Z coordinates are the same, coefficient A,
B and C are not uniquely determined, but in that case, the equation of the plane is Z = Z1 = Z2 by further adding the assumption that this plane is also perpendicular to the Z axis. For a pixel included in the pixel area 301, the coordinate calculation unit 108 obtains a Z coordinate value obtained by substituting the pixel position (X, Y) of the pixel into the equation of the plane determined as described above. , And output as depth data 109 for that pixel.

Further, the pixel area 3 corresponding to the subject 204
For the pixel included in 02, the coordinates (X
4, Y4) and the distance (Z4) to the point 208 measured by the distance measuring unit 106, the coordinates (X5, Y5) obtained from the pixel position in the image of the pixel 307 corresponding to the point 209, and the distance measuring unit. The distance to the point 209 measured at 106 (Z
From 5), the value of the Z coordinate obtained by substituting the pixel position (X, Y) of the pixel into the equation of the plane determined similarly to the case of the subject 203 is output as the depth data 109 for that pixel. To do.

Any of the pixel regions 301 and 30
For the pixels 309 in the area not included in 2, the predetermined constant value (usually the maximum value indicating infinity) is output as the depth data 109.

In addition, in the image input device in the third mode, the distance measuring unit 106 uses the rangefinder 202 to detect three points 205, 206 and 207 on the subject 203 and three points 208, 209 on the subject 204. Measure the distance to the 210 along the Z-axis. The distance calculation unit 108 performs the following interpolation / interpolation for each of the subjects 203 and 204 according to the distance data 107 input from the distance measurement unit 106.
The distance is calculated by extrapolation. Since the calculation method is common to the subjects 203 and 204, only the subject 203 will be described.

First, assuming that the object 203 is a plane, the equation of this plane is given by AX + BZ + CZ + D = 0. Coordinates (X1, Y1) obtained from the pixel position in the image of the pixel 303 corresponding to the point 205 and the distance (Z1) to the point 205 measured by the distance measuring unit 106,
The coordinates (X2, Y2) obtained from the pixel position in the image of the pixel 304 corresponding to the point 206, the distance to the point 206 (Z2) measured by the distance measuring unit 106, and the pixel 305 corresponding to the point 207. The coordinates (X3, Y3) obtained from the pixel position in the image and the point 2 measured by the distance measuring unit 106.
From the distance (Z3) to 07, three points (X1, Y1, Z
1), (X2, Y2, Z2) and (X3, Y3, Z
Assuming that 3) is a point on this plane, the coefficients A, B, C, D of the equation of the plane are obtained. Unless these three points are on the same straight line, the coefficients A, B, C and D are uniquely determined.

When these three points are on the same straight line, the coefficients A, B, C and D are not uniquely determined, but in this case, this plane is also perpendicular to the XZ plane. The equation of the plane is determined by further adding the assumption. Furthermore, if the equation of the plane cannot be determined even if this assumption is added, that is, if these three points are on the same straight line perpendicular to the XZ plane, then it is assumed that this plane is also perpendicular to the Z axis. By further adding
= Z1 = Z2 = Z3. For a pixel included in the pixel area 301, the coordinate calculation unit 108 obtains a Z coordinate value obtained by substituting the pixel position (X, Y) of the pixel into the equation of the plane determined as described above. , And output as depth data 109 for that pixel.

Further, the pixel area 3 corresponding to the subject 204
For the pixel included in 02, the coordinates (X
4, Y4) and the distance (Z4) to the point 208 measured by the distance measuring unit 106, the coordinates (X5, Y5) obtained from the pixel position in the image of the pixel 307 corresponding to the point 209, and the distance measuring unit. The distance to the point 209 measured at 106 (Z
5), the coordinates (X6, Y6) obtained from the pixel position in the image of the pixel 308 corresponding to the point 210, and the distance measuring unit 1
From the distance (Z6) to the point 210 measured at 06, the pixel position (X, Y) of the pixel is substituted into the equation of the plane determined in the same manner as in the case of the subject 203, and the Z coordinate is obtained. The value is output as the depth data 109 for that pixel.

Any of the pixel regions 301 and 30
For the pixels 309 in the area not included in 2, the predetermined constant value (usually the maximum value indicating infinity) is output as the depth data 109.

(Embodiment 2) FIG. 4 is a block diagram showing Embodiment 2 of the image input apparatus of the present invention (claim 4). Figure 4
In the figure, 401 is one or a plurality of subjects to be image input targets, 402 is an imaging unit including a camera, 4
03 denotes pixel data of an image captured by the image capturing unit 402, 4
Reference numeral 04 denotes an image processing unit that extracts a pixel area corresponding to the instructed one or more subjects 401 from the pixel data 403, and 405 indicates which pixel is the pixel of which subject 401, or for any subject 401. A signal indicating whether the pixel is not a corresponding pixel, 406 is a distance measuring unit that measures the distance to one or three points on each subject 401 for each one or more subjects 401 to be extracted,
407 is distance data measured by the distance measuring unit 406, 4
Reference numeral 08 denotes a distance calculation unit that calculates depth data corresponding to each subject according to the signal 405 and outputs the depth data in association with the pixel data 403. Reference numeral 409 denotes depth data output by the distance calculation unit 408.

In FIG. 4, the function of each component except the distance measuring unit 406 is the same as that of the component of the same name in FIG. 1 of the first embodiment. The distance measuring unit 406 is the image capturing unit 40.
The distance is calculated by using the pixel data of the image input in 2 as input.

(Embodiment 3) FIG. 5 shows the present invention (claims 5 to 5).
It is a block diagram which shows one Example of the image synthesizing system using the image synthesizing method of 7). In FIG. 5, reference numerals 501 to 503 denote an arbitrary number (N) of two or more image input devices, 504 to 506 depth data for each pixel of the image input by the image input devices 501 to 503, and 507. Is a depth comparison means, 508 is a signal for identifying which image input device the pixel determined to be the closest by the depth comparison means 507 is a pixel of the image, 509 is the closest by the depth comparison means 507, That is, the depth data 510 to 512 of the pixels determined to have the smallest depth data are the image input device 5.
Pixel data input by 01 to 503, 513
Is pixel data of the combined image which is selected and output by the pixel selection unit 513 for selecting the pixel of the corresponding image according to the identification signal 508.

In the first image synthesizing method of this embodiment, at least one of the N image input devices 501 to 503 is constituted by the image input device of the first or second embodiment.

In the second image synthesizing method, N
At least one of the image input devices 501 to 503
An image which is configured by the image input device of the first or second embodiment, and at least one of the other image input devices 501 to 503 outputs pixel data and depth data generated by calculation based on numerical data. It consists of an input device.

[0034]

As described above, according to the present invention, it is possible to add depth data for each pixel to an image input from a camera or the like, and to have depth data input from another image input device. Images and images with depth data generated by 3D computer graphics can be combined while preserving the positional relationship between the subjects included in each image, so the visual expression power in video production such as movies and television It is expected that the degree of freedom will be increased and the application of computer graphics images will be expanded.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an image input device according to a first embodiment of the present invention.

FIG. 2 is an explanatory diagram of the image input device according to the first embodiment of the present invention.

FIG. 3 is another explanatory diagram of the image input device according to the first embodiment of the present invention.

FIG. 4 is a block diagram showing a configuration of an image input device according to a second embodiment of the present invention.

FIG. 5 is a block diagram showing the configuration of an image composition system according to a third embodiment of the present invention.

FIG. 6 is a block diagram showing a configuration of a conventional image synthesizing device.

FIG. 7 (a) shows an image of a building in front of and a building at the back of a conventional image synthesizing device (b) shows an image of a car (c) shows objects in two images intertwined with each other Figure showing an image that goes back and forth

[Explanation of symbols]

 101 Subject 102 Imaging Unit 103 Pixel Data 104 Image Processing Unit 105 Pixel Identification Signal 106 Distance Measuring Unit 107 Distance Data 108 Distance Calculation Unit 109 Depth Data 201 Camera 202 Distance Meter 203, 204 Subject 205 to 210 Point on Subject 301, 302 Subject Pixel regions corresponding to the pixels 303 to 308 pixels corresponding to the points of the subject 309 pixel regions not corresponding to the subject 401 subject 402 image capturing unit 403 pixel data 404 image processing unit 405 pixel identification signal 406 distance measuring unit 407 distance data 408 distance calculating unit 409 Depth data 501-503 Image input device 504-506 Depth data 507 Depth comparison means 508 Pixel selection signal 509 Depth data 510-512 Pixel data 513 Pixel selection means 514 Pixel data

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Satoshi Nagamine Satoshi Nagamine 1006 Kadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd.

Claims (6)

[Claims] 1. An image pickup section for picking up an image and outputting pixel data of the image, an image processing section for extracting a pixel region of one or more subjects from the pixel data of the image output by the image pickup section, and the image pickup The distance along the optical axis direction of the image pickup unit from the image pickup unit to one point on the subject designated for each one or more subjects to be extracted by the image processing unit, for each subject. The distance measurement unit that performs the measurement, and the value measured by the distance measurement unit is given as the depth data common to all pixel data of individual subjects output by the image processing unit and output, and with respect to the pixel data of other regions. An image input device, comprising: a distance calculator that outputs a predetermined constant value as depth data. 2. An image pickup section for picking up an image and outputting pixel data of the image, an image processing section for extracting a pixel area of one or more objects from the pixel data of the image output by the image pickup section, and the image pickup The distance along the optical axis direction of the image pickup unit from the image pickup unit to two points on the subject specified by the image processing unit for one or more subjects to be extracted for each subject. The distance measuring unit and each subject to be extracted by the image processing unit are assumed to be planes perpendicular to the floor surface, and the distances to the pixels corresponding to the two measured points are interpolated or extrapolated. The depth of each pixel on the plane is calculated, and the value of the calculated distance is added to the pixel data of each subject output by the image processing unit and output, with respect to the pixel data of other regions. Is the depth data An image input apparatus characterized by comprising a distance calculation unit to output the grant to. 3. An image pickup section for picking up an image and outputting pixel data of the image, an image processing section for extracting a pixel area of one or more objects from the pixel data of the image output by the image pickup section, and the image pickup The distance along the optical axis direction of the image pickup unit from each unit to three points on the subject designated for each one or more subjects to be extracted by the image processing unit, for each subject. Assuming that the distance measuring unit and each subject to be extracted by the image processing unit are planes, the distances to the pixels corresponding to the three measured points are interpolated or extrapolated, and The depth of each pixel is calculated, the calculated distance value is added to the pixel data of each subject output by the image processing unit, and the pixel data is output. Value is output as depth data An image input apparatus characterized by comprising a that the distance calculator. 4. The image input device according to claim 1, wherein the distance measuring unit calculates the distance by using pixel data of an image output by the image pickup unit. 5. Pixel data and depth data of an image output by the image input device according to claim 1, are provided as at least one input, and have pixel data and depth data for each pixel. One or more images are taken as another input, and the depth data is compared for each pixel at the corresponding pixel positions of all the input images, and the pixel closest to the image pickup unit, that is, the pixel having the smallest depth data is selected. An image synthesizing method characterized by outputting as a pixel at that pixel position of a synthesized image. 6. The at least one of the other input images comprises pixel data and depth data generated by calculation based on numerical data.
The described image composition method.