JP2888108B2 - Parallax image creation method and device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for creating a parallax image of a subject, and more particularly, to a method for creating a parallax image, in which a three-dimensional shape of a subject is input to a background portion and a subject portion. The present invention relates to a parallax image creating method and apparatus capable of separating a background and attaching a background to a plane arranged behind a subject, thereby reducing errors in the background part and creating a parallax image with high accuracy.

[0002]

2. Description of the Related Art In recent years, various methods for displaying a three-dimensional image using a hologram and other methods have been proposed. Among these, the parallax panoramagram method of displaying a stereoscopic image by combining a large number of parallax images can observe different images depending on the observation position,
Since a natural three-dimensional effect can be obtained, it has been applied to holographic stereograms and stereoscopic vision using a lenticular lens, and has received a very excellent evaluation.

However, in such a method, taking a holographic stereogram as an example, in order to obtain a natural stereoscopic effect, the number of parallax images must be as follows.
A large number of 50 to 100 parallax images are required.
When the number of the parallax images is large, the shooting of the parallax image becomes large, and it takes a long time to shoot, and the size of the shooting device is also large. For this reason, it is very difficult to shoot a fast moving subject or a large subject such as a building.

Therefore, it is conceivable to obtain a three-dimensional shape of a subject from a small number of parallax images and obtain a new parallax image based on the three-dimensional shape. The conventional stereo image method, which is one of the methods, has the following various problems in creating a parallax image.

That is, in the three-dimensional shape measurement by the stereo image method, a continuous stereo image (Epipolar-P) is used.
When a three-dimensional shape of a subject is obtained from the lane image (EPI), a camera that generally captures the subject performs the shooting such that the subject is located at the center of the screen.
For this reason, a background having depth away from the subject is captured only in a part of the parallax images. Further, since the background is blocked by the subject, a part of the background may not be photographed on the image.

As described above, since the background portion has a smaller amount of information than the subject portion, it is difficult to obtain a correct depth only from a continuous image, which causes an error in the background portion.

[0007]

As described above, in the method of creating a parallax image by the conventional stereo image method, since the information amount of the background portion is smaller than the information amount of the subject portion, only the continuous image is generated. It is difficult to obtain the correct depth from the image, and there is a problem that an error occurs in the background.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problem, and has an extremely reliable parallax image creating method capable of accurately creating a parallax image by reducing background errors. It is intended to provide a device.

[0009]

In order to achieve the above object, according to the first aspect of the present invention, in a method for creating a parallax image of a subject, first, a parallax image pair having a small parallax is set differently. A plurality of images are taken from the position, then a parallax image of the subject and a parallax image of the background are separated from the parallax image pair, a continuous stereo image of the subject is created from the parallax image of the subject, and then a continuous stereo image is formed. The three-dimensional distribution of the subject is measured by decomposing the subject into directional components, and three-dimensional voxel data of the subject is generated.
By pasting the parallax image of the background portion on the plane arranged behind the three-dimensional voxel data, three-dimensional voxel data of the subject in consideration of the background portion is generated, and then the three-dimensional voxel data of the subject in consideration of the background portion is generated. By projecting onto a two-dimensional plane in an arbitrary direction, a new parallax image in an arbitrary direction is created.

According to a second aspect of the present invention, in a device for creating a parallax image of a subject, a photographing means for photographing a plurality of parallax image pairs having minute parallax from different positions, and a parallax photographed by the photographing means. Parallax image separating means for separating the parallax image of the subject portion and the parallax image of the background portion from the image pair, and continuous stereo image generation for forming a continuous stereo image of the subject from the parallax images of the subject portion separated by the parallax image separating means Means for measuring the three-dimensional distribution of the subject by decomposing the continuous stereo image created by the continuous stereo image creating means into directional components, and generating three-dimensional voxel data of the subject; The background portion is viewed on a plane arranged behind the three-dimensional voxel data of the subject created by the three-dimensional voxel data generating means. A background parallax image pasting unit that attaches an image and generates three-dimensional voxel data of the subject in consideration of the background part, and three-dimensional voxel data of the subject in consideration of the background part generated by the background parallax image pasting unit Is projected onto a two-dimensional plane in an arbitrary direction to create a new parallax image in an arbitrary direction.

[0011]

Therefore, in the parallax image creating method and apparatus according to the present invention, a plurality of parallax image pairs having minute parallax are photographed from different positions, and a parallax image of a subject portion and a parallax image of a background portion are obtained from the parallax image pairs. Is separated, a continuous stereo image of the subject is created from the parallax image of the subject portion, and the continuous stereo image is decomposed into directional components to measure the three-dimensional distribution of the subject, thereby generating three-dimensional voxel data of the subject. By attaching the parallax image of the background to the plane disposed behind the three-dimensional voxel data of the subject, three-dimensional voxel data of the subject taking the background into account is generated. By projecting the dimensional voxel data on a two-dimensional plane in an arbitrary direction, a new parallax image in an arbitrary direction is created.

Thus, when the three-dimensional shape of the subject is input, the background portion and the subject portion are separately input, and the background is attached to a plane arranged at the rear portion of the subject, thereby reducing errors in the background portion. Thus, a parallax image can be created with high accuracy.

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS First, the concept of the present invention will be described.

The continuous stereo image is obtained by arranging horizontal cutting lines of each parallax image in the order of photographing, and the trajectory of the corresponding point of the parallax image on the continuous stereo image has an inclination corresponding to the distance from the camera. Appears as a straight line.

Therefore, the continuous stereo image is decomposed into directional components, the depth of a portion having no occlusion is obtained, and the process of obtaining the depth again from the continuous stereo image excluding the portion where the depth is determined is repeated. , The depth data is reconstructed to generate three-dimensional voxel data, and the three-dimensional voxel data is projected onto a two-dimensional plane in an arbitrary direction, thereby forming a new parallax image in an arbitrary direction. Can be created.

Further, in the present invention, when a parallax image of a subject is photographed by a camera, a parallax image is photographed as a plurality of parallax image pairs of a small parallax as shown in FIG. Then, using each pair of the small parallax images,
The subject part and the background part are separated from the parallax image. The subject portion is determined from the continuous stereo image by using a predetermined method.
Find the dimensional shape. The background portion is attached to a plane behind the subject portion. In this way, a new parallax image in an arbitrary direction can be obtained by projecting the created three-dimensional shape onto a two-dimensional plane in a computer.

On the other hand, when the depth of an object is obtained from a pair of minute parallax images, a phenomenon (occlusion) occurs in which an object in the foreground obscures an object in the back because the parallax is small.
It is rare that the correct depth cannot be obtained. for that reason,
Depth can be obtained relatively easily from two parallax image pairs. However, since the parallax is small, the accuracy of the depth is conversely inferior. Therefore, the three-dimensional shape of the object created using information only from the pair of minute parallax images is
It is not suitable as parallax image creation means.

Therefore, in view of the above, the present invention combines a depth measurement method from a pair of minute parallax images and a depth measurement method from a continuous stereo image to provide a three-dimensional shape measurement method suitable as a parallax image generating means. Is provided.

Hereinafter, an embodiment of the present invention based on the above concept will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing an example of the overall configuration of a parallax image creating apparatus according to the present invention. That is, as shown in FIG. 1, the parallax image creating apparatus according to the present embodiment includes a plurality of photographing units (4 in FIG. 1) arranged at regular intervals.
) Cameras 11, 12, 13, 14 and each camera 1
Parallax image separation means 21, 22, 23, 24 provided corresponding to 1, 12, 13, 14; continuous stereo image creation means 3, depth determination means 4, three-dimensional voxel data generation means 5, It comprises a background parallax image sticking means 6 and a parallax image creating means 7.

Here, the cameras 11, 12, 13, 14
Indicates that when capturing a parallax image of the subject 8, a plurality of parallax image pairs having minute parallax from different positions (4 in FIG.
Photos).

Also, the parallax image separating means 21, 22, 2,
Reference numerals 3 and 24 denote parallax images of the subject 8 and the background 9 from parallax image pairs captured by the cameras 11, 12, 13 and 14.
This is for separating the parallax image of the part.

Further, the continuous stereo image creating means 3
From the parallax images of the four subjects 8 separated by the parallax image separating means 21, 22, 23, and 24, the cutting lines in the horizontal direction of each of the parallax images are arranged in the shooting order, whereby a continuous stereo image of the subject 8 is formed. To create.

Further, the depth determining means 4 measures the three-dimensional distribution of the subject 8 by decomposing the continuous stereo image created by the continuous stereo image creating means 3 into directional components, and determines the depth of the portion without occlusion. Is calculated, and the depth of the occlusion portion is calculated by repeating the process of calculating the depth again from the continuous stereo image excluding the portion where the depth is determined.

In this case, as a method of determining the depth by decomposing the directional components of the continuous stereo image, for example, processing is performed by a method using the variance of the density of a straight line on the continuous stereo image. This variance method is a process of determining the variance of the density on various straight lines on a continuous stereo image and determining that an object exists at a depth corresponding to the slope of the straight line when the variance is below a certain reference value. Is the way.

On the other hand, three-dimensional voxel data generating means 5
Is for reconstructing the depth data calculated by the depth determining means 4 to generate three-dimensional voxel data of the subject 8.

The background parallax image pasting means 6 includes a subject 8 generated by the three-dimensional voxel data generating means 5.
The nine-dimensional parallax image of the background is attached to a plane disposed behind the three-dimensional voxel data of the above (3) to generate three-dimensional voxel data of the subject in consideration of the nine backgrounds.

Further, the parallax image creating means 7 projects the three-dimensional voxel data of the subject in consideration of the nine backgrounds generated by the background parallax image sticking means 6 onto a two-dimensional plane in an arbitrary direction. Is to create a new parallax image in the direction of.

Next, a parallax image forming method in the parallax image forming apparatus of the present embodiment configured as described above will be described.

First, a parallax image in which the subject 8 and the background 9 are separated from the micro-parallax image pair is created by a method using a micro-parallax image pair that is somewhat inaccurate but hardly causes occlusion. Generally, the background 9 and the subject 8 are often separated from each other, and the separation of the background 9 and the subject 8 does not require much depth accuracy.

Next, a continuous stereo image is created using the four separated parallax images of the subject 8, and the shape of the subject 8 is determined.

Next, a plane is set at an appropriate position behind the determined shape of the subject 8, and the background 9 determined from the pair of minute parallax images is attached.

Thereafter, the three-dimensional shape obtained in consideration of the background 9 thus obtained is re-projected on a two-dimensional plane by a computer or the like, whereby a parallax image in an arbitrary direction can be created.

Hereinafter, a method of creating such a parallax image will be described more specifically with reference to a flowchart shown in FIG.

First, a pair of parallax images having minute parallax is moved from different positions by the cameras 11, 12, 13, and 14.
Take a picture.

Next, the parallax image separating means 21, 22, 2
3 and 24, the parallax image of the subject 8 and the background 9 are obtained from the parallax image pair captured by the cameras 11, 12, 13, and 14.
The parallax image of the part is separated.

Next, the continuous stereo image forming means 3 converts the parallax images of the eight subjects 8 separated by the parallax image separating means 21, 22, 23, 24 from the Y-line continuous stereo of the subject 8. Create an image (step S
1. Step S2).

Next, the continuous stereo image created by the continuous stereo image creating means 3 is
By taking the variance on a straight line with various inclinations, it is decomposed into continuous stereo image direction components. Since the inclination and the depth of the straight line indicating the locus of the corresponding point correspond, the depth can be obtained. Here, the depth of the portion where occlusion exists cannot be determined because the object in front blocks the locus of the corresponding point. That is, in the first depth determination process, the depth of the portion without occlusion is determined (step S3).

Next, the part whose depth has been determined is removed from the previously created continuous stereo image by the continuous stereo image creating means 3 (step S5).

Then, the depth is determined by the depth determining means 4 again taking various variances on the straight line of the inclination. At this time, the part whose depth has already been determined (the part removed by the above processing) is not used for the variance calculation. With this processing, the depth of the part hidden by the part whose depth has been determined by the first processing can be obtained (step S3).

By repeating the processing of steps S3 and S5, the depth of the point where occlusion occurs can be obtained (step S4).

By calculating these processes for all horizontal lines, the depth of the subject 8 can be obtained (steps S6 and S7).

Next, the three-dimensional voxel data generating means 5 reconstructs the depth data calculated by the depth determining means 4 to obtain three-dimensional voxel data of the subject 8.

Next, the background parallax image attaching means 6
The subject 8 generated by the three-dimensional voxel data generating means 5
The parallax image of the nine backgrounds is attached to the plane disposed behind the three-dimensional voxel data of (3) to obtain three-dimensional voxel data of the subject in consideration of the nine backgrounds (step S8).

Finally, the three-dimensional voxel data generated by the three-dimensional voxel data generating means 5 is projected by the parallax image generating means 7 onto a two-dimensional plane in an arbitrary direction by a known computer graphics technique. Thereby, a new parallax image in an arbitrary direction can be created (step S9).

In the above description, it may not be possible to determine the depth in a portion where there is no change in density. However, if a depth error occurs in a portion where there is no change in density, a pixel having the same density is displayed on the parallax image. Are simply replaced, the image quality is not degraded at all, and there is no particular problem.

FIG. 5 is a schematic diagram showing an example of a case where a new parallax image is created from a small number (four) of parallax images by the parallax image creating apparatus of this embodiment.

Next, a method of separating the background 9 and the subject 8 from the pair of minute parallax images will be described more specifically.

Various methods have been proposed for obtaining the depth of an object from a pair of small parallax images. Here, a method using the difference between parallax images is described.
This will be described specifically.

That is, specifically, the density at one coordinate x, y of a parallax image pair having parallax only in the x direction is represented by I ₀ (x, y), and the coordinates x, y of the other parallax image.
At a concentration of I ₁ (x, y).

Assuming that the partial differential value in the x direction on the parallax image is I _x and the partial differential value of x between the parallax images is I _vx , I _x = I ₀ (x + 1, y) −I ₀ (x, y ) I _vx = I ₁ (x, y) −I ₀ (x, y) At this time, the movement amount h of the corresponding point at the coordinates x, y can be obtained by the following equation.

H = I _x · I _vx / I _x ^{2 In} order to eliminate the influence of noise on the parallax image, averaging over a certain area P gives

(Equation 1) Becomes Since the movement amount h corresponds to the depth of the object, the depth at the coordinates x and y can be obtained.

Thus, the background 9 and the subject 8 can be separated from the pair of minute parallax images using the depth obtained in this manner.

Next, a method for obtaining the depth of the occlusion portion will be described more specifically with reference to FIGS.

Specifically, as shown in FIG. 3, a continuous stereo image is scanned with straight lines having various inclinations, and the variance on the straight lines is obtained. When the variance is equal to or less than a certain reference value, it can be considered that an object exists at a depth corresponding to the direction of the straight line.

In this case, the depth of the occlusion portion cannot be determined because the locus of the corresponding point is interrupted, and the density change at the interrupted portion becomes large. Therefore, as shown in FIG. 4, the portion excluding the region whose depth is determined in the above-described processing from the continuous stereo image is again scanned with various straight lines, and the region hidden by the portion whose depth has already been determined. Can be determined.

As described above, the parallax image creating apparatus of the present embodiment, when capturing a parallax image of the subject 8, captures a plurality of parallax image pairs having minute parallax from different positions (four in the drawing). Image separation means 21 for separating a parallax image of an object 8 and a parallax image of a background 9 from a pair of parallax images captured by the cameras 11, 12, 13, 14 and parallax images taken by the cameras 11, 12, 13, 14. , 22,23,24
From the parallax images of the four subjects 8 separated by the parallax image separating means 21, 22, 23, and 24, the horizontal cutting lines of the respective parallax images are arranged in the photographing order, so that the continuous stereo of the subject 8 is obtained. A continuous stereo image creating unit 3 for creating an image, and a three-dimensional distribution of the subject 8 is measured by decomposing the continuous stereo image created by the continuous stereo image creating unit 3 into directional components, and the depth of a portion without occlusion is measured. And a depth determining means 4 for calculating the depth of the occlusion portion by repeating the process of calculating the depth again from the continuous stereo image excluding the portion where the depth is determined, and the depth data calculated by the depth determining means 4. Is reconstructed into 3 of the subject 8
The three-dimensional voxel data generating means 5 for generating the three-dimensional voxel data, and the parallax image of the background 9 is pasted on a plane arranged behind the three-dimensional voxel data of the subject 8 generated by the three-dimensional voxel data generating means 5. , Background 9
Background parallax image pasting means 6 for generating three-dimensional voxel data of the subject in consideration of the part, and three-dimensional voxel data of the subject considering the nine backgrounds generated by the background parallax image pasting means 6 Projected on a two-dimensional plane in the direction
And a parallax image creating means 7 for creating a new parallax image in an arbitrary direction.

Therefore, when the three-dimensional shape of the subject 8 is input, the background 9 and the subject 8 are separately input, and the background 9 is pasted on a plane arranged at the rear part of the subject 8, whereby the background 9 Since the error of the section can be reduced, a parallax image can be created with high accuracy.

Further, since the depth is determined by decomposing the continuous stereo image into directional components and giving a depth corresponding to the direction, it is not necessary to perform a feature point extraction process on the original image or the continuous stereo image.
In addition, since the corresponding points of a plurality of parallax images can be determined at once, the versatility is high and the three-dimensional shape of the subject 8 can be easily obtained.

Further, since a new parallax image in an arbitrary direction can be created by projecting the three-dimensional shape of the subject 8 onto a two-dimensional plane in an arbitrary direction, the versatility is extremely high. It is suitable for stereoscopic display in which a natural subject is assumed.

On the other hand, even if occlusion occurs,
Since the processing of the directional component decomposition of the continuous stereo image is repeatedly performed except for the part where the depth is determined, the versatility is extremely high, and the three-dimensional shape of the subject 8 can be obtained easily and accurately, and the parallax image can be obtained. This is a suitable method for creating

That is, in the conventional method, in the portion where occlusion occurs, the trajectory of the corresponding point is interrupted by the object in front and the trajectory of the corresponding point is interrupted, so that the depth of the subject cannot be obtained accurately. However, in the present embodiment, the process of decomposing the continuous stereo image into directional components and determining the depth is performed a plurality of times, and the depth is determined in order from the object in front, so that the depth of the portion where occlusion occurs is determined. It is possible to ask. In this case, since the calculation between a plurality of parallax images can be performed at once, the three-dimensional shape of the subject can be obtained by a relatively fast operation, which is a method suitable for creating a parallax image.

Further, since only a process of decomposing a continuous stereo image into directional components and calculating a depth is required, a three-dimensional image can be obtained directly from the continuous stereo image, and extremely simple processing by hardware can be performed. Becomes possible.

It should be noted that the present invention is not limited to the above embodiment, but can be implemented as follows.

(A) In the above embodiment, four cameras 1
A case has been described in which 1, 12, 13, and 14 are installed to obtain four parallax images of the subject 8, but the present invention is not limited to this.
Alternatively, four cameras may be installed, and the parallax images may be obtained by photographing the subject 8 while moving the cameras on a straight line at a constant interval (pitch).

(B) In the above-described embodiment, a case has been described in which the method of determining the depth by decomposing the directional component of the continuous stereo image uses the method of dispersing the density of straight lines on the continuous stereo image. However, the present invention is not limited to this. Instead, for example, a method based on a Fourier transform or a method based on a Hough transform may be used to decompose the direction component.

[0067]

As described above, according to the present invention, when a parallax image of a subject is created, a plurality of parallax image pairs having minute parallax are photographed from different positions, and then the parallax image pair is read from the subject portion. Separates the parallax image of the subject from the parallax image of the background part, creates a continuous stereo image of the subject from the parallax image of the subject part, and then measures the three-dimensional distribution of the subject by decomposing the continuous stereo image into directional components Then, three-dimensional voxel data of the subject is generated, and then a parallax image of the background part is pasted on a plane arranged behind the three-dimensional voxel data of the subject, thereby obtaining a three-dimensional image of the subject in consideration of the background part.
Since the three-dimensional voxel data of the subject in consideration of the background portion is generated on the two-dimensional plane in an arbitrary direction after generating the three-dimensional voxel data, a new parallax image in an arbitrary direction is created. An extremely reliable disparity image creating method and apparatus capable of creating an accurate disparity image by reducing errors in the background portion can be provided.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of a parallax image creating apparatus according to the present invention.

FIG. 2 is a flowchart for explaining the operation in the embodiment.

FIG. 3 is an exemplary diagram for explaining a method of obtaining a depth of an occlusion portion in the embodiment.

FIG. 4 is an exemplary diagram for explaining a method of obtaining a depth of an occlusion portion in the embodiment.

FIG. 5 is an exemplary diagram illustrating an example of creating a new parallax image from a small number of parallax images in the embodiment.

FIG. 6 is a schematic diagram for explaining the concept of the present invention.

[Explanation of symbols]

11, 12, 13, 14 ... Cameras 21, 22, 23,
24: parallax image separating means, 3: continuous stereo image generating means, 4: depth determining means, 5: three-dimensional voxel data generating means, 6: background parallax image sticking means, 7: parallax image generating means, 8: subject , 9 ... background.

Claims (2)

(57) [Claims] 1. A method of creating a parallax image of a subject, comprising: first capturing a plurality of parallax image pairs having a small parallax from different positions; and then generating a parallax image of a subject portion and a background portion from the parallax image pair. By separating the parallax image from the parallax image, a continuous stereo image of the subject is created from the parallax image of the subject portion, and then the three-dimensional distribution of the subject is measured by decomposing the continuous stereo image into directional components. Subject 3
Generating three-dimensional voxel data of the subject, and then pasting the parallax image of the background portion on a plane arranged behind the three-dimensional voxel data of the subject to generate three-dimensional voxel data of the subject in consideration of the background portion Thereafter, a new parallax image in an arbitrary direction is created by projecting the three-dimensional voxel data of the subject in consideration of the background onto a two-dimensional plane in an arbitrary direction. Parallax image creation method. 2. An apparatus for creating a parallax image of a subject, comprising: photographing means for photographing a plurality of parallax image pairs having minute parallax from different positions; and a parallax image of a subject portion from the parallax image pairs photographed by the photographing means. A parallax image separating unit that separates a parallax image of a subject and a parallax image of a background portion; a continuous stereo image forming unit that forms a continuous stereo image of a subject from a parallax image of a subject portion separated by the parallax image separating unit; By decomposing the continuous stereo image created by the creating means into directional components, 3
A three-dimensional voxel data generating means for measuring three-dimensional voxel data of the subject by measuring a three-dimensional distribution, and a plane arranged behind the three-dimensional voxel data of the subject created by the three-dimensional voxel data generating means; A background parallax image pasting means for pasting a parallax image of a background part to generate three-dimensional voxel data of a subject in consideration of the background part, and a background part generated by the background parallax image pasting means. A parallax image generating unit configured to project three-dimensional voxel data of a subject onto a two-dimensional plane in an arbitrary direction to generate a new parallax image in an arbitrary direction.