JP2004200973A - Apparatus and method of inputting simple stereoscopic image, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the conventional problem that two cameras must be usually used to input a stereoscopic image, and the size and cost of the cameras cause a barrier against spreading this. <P>SOLUTION: A plurality of images are inputted into a photographing object under different illumination conditions. As the illumination conditions, "light up" and "no light" are set, and an image having a larger illuminance difference when computing the differential between images under each condition is determined to be close to illumination. Although a rough depth value is calculated at this time, this value is divided into a plurality of depth levels, parallax equivalent to a depth value is assigned to each level, and the parallax between respective levels is changed so that a photographing object in a scenery state can be recognized as one object in order to prevent a stereoscopic image from being perceived as a scenery. By performing interpolation processing using image information of a depth level farther than an adjacent depth level for the omission of data in a stereoscopic image generated due to the addition of parallax, information about shielding of an object can be faithfully reproduced, and a natural stereoscopic image can be generated. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a simple stereo image input device, a method, a program, and a recording medium that generate a stereo image from a plurality of images captured by changing the presence or absence of illumination or the intensity of a captured object.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, as a natural image input device for a stereo image display device, two or more cameras are arranged in parallel or with a slight convergence so that their optical axes intersect with each other, and the photographed images are left and right as they are. A method of inputting a display image to a display device has been used (see Non-Patent Document 1). Although this is the simplest configuration, it is more expensive because it requires the use of multiple cameras. Further, arranging a plurality of cameras in a limited size such as for a mobile phone is disadvantageous in terms of miniaturization.
[0003]
A method of generating a stereo image from a two-dimensional “moving” image captured by one camera has also been proposed (see Patent Document 1). In this method, one of the left and right eye images is input as a two-dimensional input moving image as it is, and the other eye image is delayed as a frame delay according to the magnitude of the motion vector of the input two-dimensional moving image. This is a method of stereoscopically viewing by changing the amount and inputting. In this method, since a stereo image is generated using motion information of a two-dimensional moving image, when a plurality of objects move in different directions, the parallax between the objects is reversed, and the stereoscopic image is disturbed. There is.
[0004]
On the other hand, a method of generating a stereo image from a two-dimensional “still” image has been proposed (see Patent Document 2). In this method, since a stereo image is generated from one still image, there is a shortcoming that information on distance is insufficient and the image quality level of the generated stereo image is low.
[0005]
[Patent Document 1]
Patent No. 2594235 (Method for converting 2D video to 3D video and 3D video generation device)
[Patent Document 2]
Patent No. 3067097 (3D image data generation method)
[Non-patent document 1]
Supervised by Takehiro Izumi, edited by NHK Science and Technical Research Laboratories, "Basics of 3D Video", Ohmsha, June 5, 1995, pp. 71-73
[0006]
[Problems to be solved by the invention]
Currently, mobile phones with a camera, notebook PCs, PDAs, and the like are being actively developed. Normally, images taken by a camera attached to these are only two-dimensional information. However, if the acquisition and generation of distance information becomes possible, the range of applications such as stereo image display is expanded. However, up to now, it has been necessary to use two cameras in order to input a stereo image, and the size and the price have been barriers to popularization.
[0007]
[Means for Solving the Problems]
SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problem, and makes it possible to generate a stereo image easily with a small size.
[0008]
The stereo image input device of the present invention includes: an image recording unit that records a plurality of images; and a depth value calculation unit that calculates a depth value of a shooting target based on a comparison operation of pixel values of corresponding pixels in the plurality of images. And a stereo image generating means for generating a stereo image based on the depth value obtained by the depth value calculating means.
[0009]
The stereo image input device according to the present invention operates as follows. A plurality of images are input to a shooting target under different lighting conditions. The depth value is calculated by comparing the plurality of images. As a comparison method, for example, "illuminated" and "non-illuminated" are set as illumination conditions, and when the difference between images under each condition is calculated, the greater the luminance difference, the closer to illumination.
[0010]
The depth value calculated at this time is coarse due to the unevenness of the illumination light and the reflection characteristics of the object to be photographed. However, the depth value is divided into a plurality of depth levels, and a parallax corresponding to the depth value is assigned to each level. In order to prevent the stereoscopic image from being perceived as described above, the parallax between the levels is changed so that the shooting target in the divided state is recognized as one object.
[0011]
Furthermore, information about occlusion of an object is more faithfully obtained by performing interpolation processing on data loss in a stereo image caused by the addition of parallax using image information at an adjacent and farther depth level. And a more natural stereo image can be generated.
[0012]
In this method, only one photographing device is required, so that the size can be reduced as compared with a case where a plurality of photographing devices are used. In addition, even in the case of detection with coarse accuracy, the parallax is changed by image processing using the approximate distance information of the object, and data loss processing is performed by generating a stereo image, thereby avoiding the perception of writing and observing. It is possible to present a high quality stereoscopic image to the user.
[0013]
BEST MODE FOR CARRYING OUT THE INVENTION
FIG. 1 shows a simplified stereo image input device according to an embodiment of the present invention. The simple stereo image input device is connected to a photographing device 1-1 having or connected to a lighting device 1-2 for illuminating a photographing target, and an image recording device 1-3 for capturing and holding image data of a photographing target, and the intensity of illumination. Calculates approximate distance information of the subject by comparing a plurality of images of the photographing targets having different image data, and generates a stereo image of the photographing target from the result, and a system control device 1 that controls the above units. 5 is comprised.
[0014]
Next, an outline of an image input method according to an embodiment of the present invention will be described with reference to FIG. (A) Shoot with flash. The face image (foreground) is photographed brightly, but the background region that is far from the camera is photographed dark because the light of the flash does not reach. (B) Shoot the same image without flash. Both the face image (foreground) and the background area are shot dark. (C) Subtract the gray value of the image with flash and the image without flash. As a result of the subtraction, an image of a portion having a difference in brightness is obtained. Since a portion having a large difference in brightness is close to the illumination and a portion having a small difference in brightness is far from the illumination, rough distance information between the foreground and the background can be obtained. (D) Binarization is performed to absorb the effects of the color and brightness of the subject, and binarized distance information is obtained. In the figure, black is the foreground and white is the background. (E) Using the binarized distance information, obtain a face image (foreground) of the captured video, assign parallax thereto, and create a right-eye image and a left-eye image. However, in this state, the face image looks flat (split), so that (F) the face images for the right eye and the left eye are rounded and processed to look natural. As a result, a stereo image that looks natural can be obtained.
[0015]
Hereinafter, the operation of the simple stereo image input device will be described with reference to the flowchart of the simple stereo image input method shown in FIG. When a stereo image is easily generated for a certain photographing target, first, an illumination condition and the number k of photographed images are input (step 3-1). The lighting conditions and the number of shots may be determined in advance. Next, the object to be photographed is illuminated using the illumination device 1-2 under the j-th illumination condition, and an image captured by the imaging device 1-1 is input (step 3-2). The illumination condition is determined by the intensity of the illumination, and illumination is performed in accordance with an illumination signal from the imaging device 1-1 (for example, pulsed light synchronized with an imaging shutter). The j-th input image signal is stored as image data in the frame memory j of the image recording device 1-3 (step 3-3). The processing from step 3-2 to step 3-3 is repeated until the number of images reaches k (steps 3-4 and 3-5). In this embodiment, since the number of times of image capturing is assumed to be two (k = 2), the number of frame memories (1-31, 1-32) on the image recording means 1-3 is two. The frame memory may be increased accordingly. When the input of the image is completed, the image data photographed by the command of the system control device 1-5 is transferred from the frame memory to the image operation portion 1-41 of the operation device 1-4 (step 3-6).
[0016]
The image calculation unit 1-41 outputs color information (two-dimensional image) of appropriate brightness using a plurality of images (3-7). Further, the image calculation unit 1-41 compares a plurality of images having different lighting conditions, generates rough distance information from a difference in brightness between the plurality of images, and outputs the information to the stereo image generation unit 1-42 (3). -8). For the calculation of the distance information, image difference calculation, division, binarization, edge detection, and the like can be used. Examples of the color information synthesizing process include a method of converting each image so as to maximize the contrast and extracting an image having the highest contrast, and a method of synthesizing them with appropriate weighting. A combination of these processes may be selected by an instruction from the system controller 1-5.
[0017]
FIG. 4 is an example of a flowchart showing the procedure for calculating distance information. Here, the depth value is two depths before and after. Two lighting conditions, “lighting on” and “lighting off,” are set, and the number of shots is two. At this time, the pixel value of the i-th pixel of the image data when "illuminated" is Ai, the pixel value of the i-th pixel of the image data when "no illumination" is Bi, and the binarized data is Di. Expressed by When calculating the depth value, first, a threshold value n and an image size s are determined (step 4-1). The threshold value n can be arbitrarily determined by a parameter for dividing an image into binary values. The image size s is a value obtained by multiplying the horizontal pixel size x and the vertical pixel size y of the input image data. Next, the difference between the pixel values Ai and Bi is calculated, and if it is equal to or larger than the threshold value n, Di is set to 1, otherwise 0 is set (steps 4-2 to 4-4). The processing of steps 4-2 to 4-4 is repeated for all pixels of the image data (steps 4-5 to 4-6). The pixel whose value of the data Di after binarization is 1 is determined to be close to the image capturing apparatus, and the other pixels are determined to be near and far from the image capturing apparatus. In step 4-2, Di may be set to 1 if (Ai-Bi) / Bi is larger than a predetermined ratio, and may be set to 0 otherwise. Also, even when data of a plurality of images to be compared are shifted, by taking an average value of several neighboring pixels when comparing the pixel i, it is possible to reduce the influence of the shift and to cope with a moving image. It becomes.
[0018]
In step 3-9 in FIG. 3, the stereo image generation unit 1-42 calculates a depth value for actually outputting as a stereo image using the rough distance information obtained by the image calculation unit 1-41. . Here, first, the amount of projection L of the subject in the stereo image finally output (displayed) is specified, and the depth of each pixel in the image is determined so as to be observed at this distance. Specifically, first, when the distance information is divided into two types (binarization) of the foreground and the background, the maximum projection amount L of the foreground is set. However, in this state, the foreground is observed as an object having a small thickness such as writing. To prevent this, as shown in FIG. 5, for each of the right-eye image and left-eye image in the split state shown in FIG. , (C) calculates the distance between the center of gravity and each pixel in the foreground, and (D) performs a process of setting the pixels in the foreground area that are farther away from the center of gravity into the far view. This distance is determined by a method of interpolating a distance to a distant view with a quadratic curve, a method of interpolating with a straight line, a maximum projection amount L up to, for example, a length of 30% centered on the center of gravity, and a distance from that to the boundary of the foreground. It is possible to set a method of performing interpolation using a quadratic surface, a method of arbitrarily specifying the thickness of an object, and the like. (E) is a display image.
[0019]
For a distant view, as shown in FIG. 6, the depth value is determined so that the distant view is observed farther than the foreground object. Specifically, if the background image (left and right eyes) shown in (A) is arranged on the back side of the screen, as shown in (B), the image for the left eye is moved further leftward as shown in (B). By moving it and moving it further to the right in the image for the right eye, it is possible to display it on the far side of the screen. The processing related to the depth determination of the foreground and the background can be included in the same processing as the processing of 3-10 described below.
[0020]
The depth value set in step 3-9 in FIG. 3 is converted into parallax information on the image in step 3-10. Specifically, in order to observe the stereo image at the depth specified in step 3-9, the depth value calculated from the distance between the eyes and the size of one pixel in the image by the principle of triangulation is the same. Thus, the parallax of the stereo image is determined.
[0021]
At step 3-11, a stereo image is actually generated based on the parallax determined at step 3-10. Specifically, as shown in FIG. 7, first, (A) a two-dimensional image output at 1-41 as an original image of a stereo image (image for the left and right eyes) or 1-31 and 1-32 Prepare an image to be output with. Next, (B) the parallax amount corresponding to each pixel on the screen is moved in the left-right direction in the left-right image according to the specified parallax amount. For example, when one point in the image has depth information of the parallax d in the front direction of the screen, the pixel value is moved to the left direction of the image by d / 2 for the right eye, and an image is generated. For the left eye, an image is generated by moving the image to the right by d / 2. At this time, those having a parallax of 0 are copied as they are to the left and right eye images. With this method, a stereo image can be generated for data in which there is a corresponding point between the left and right images. However, there are cases where pixel values (color data) do not exist in the left and right eye images generated here. I do. This is processed in the next step 3-12.
[0022]
For a pixel having no color data, as shown in FIG. 8 (A), data corresponding to a farther distant pixel whose parallax is not lost in the left-right direction of the pixel is compared with the left-right direction. (Example 1). In addition, a method in which an area where no color data exists is replaced with color data of an adjacent area having a parallax that is farther away (Example 2). As a result, when the foreground blocks the background, even the region that appears near the boundary between the foreground and the background in the left-right direction and is visible only to one eye is located far away from the foreground by interpolating using the color data of the background region. Since it is given the information that the stereo image is being reproduced, a somewhat natural stereo image can be generated (FIG. 8B).
[0023]
As described above, a stereo image can be generated.
[0024]
In order to generate the above-mentioned stereo image, it is not necessary to use a plurality of cameras, and since the lighting device can be arranged at a position adjacent to the camera, it is possible to reduce the size of the conventional device, and It is also possible to mount it on such as
[0025]
The device of the present invention can also be realized by a computer program, and the program can be recorded on a recording medium or provided through a network.
[0026]
As described above, the invention made by the inventor has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and can be variously modified without departing from the gist of the invention. Of course.
[0027]
【The invention's effect】
As described above, according to the present invention, the following effects are produced.
According to the first aspect of the present invention, since there is no need to use only one photographing device and a plurality of photographing devices, it is possible to reduce the size as compared with the conventional stereo image photographing device.
According to the second aspect of the present invention, by setting the result of the simple binarization process to roughly two distances, it is hard to be affected by unevenness of illumination light and the reflection characteristic of the photographing target, and the calculation load is small. High-speed processing becomes possible.
According to the invention of claim 3, it is possible to easily generate a stereo image from which a natural depth perception can be obtained without performing complicated calculation processing.
[Brief description of the drawings]
FIG. 1 is a diagram showing a simple stereo image input device according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating an outline of an image input method according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating an example of a flowchart of a simple stereo image input method.
FIG. 4 is a diagram illustrating an example of a flowchart illustrating a rough distance information calculation procedure.
FIG. 5 is a diagram illustrating a write split avoidance process;
FIG. 6 is a diagram illustrating a method of determining a depth value of a background portion.
FIG. 7 is a diagram illustrating a stereo image generation method.
FIG. 8 is a diagram illustrating a method of interpolating missing color data caused by stereo image generation.
[Explanation of symbols]
1-1: imaging device, 1-2: lighting device, 1-3: image recording device, 1-4: arithmetic device, 1-5: system control device, 1-11: solid-state image sensor, 1-12: A / D conversion unit, 1-31 ... frame memory 1, 1-32 ... frame memory 2, 1-41 ... image operation unit, 1-42 ... stereo image generation unit, 3-1-3-12 ... simple stereo image input Each step of the procedure, 4-1 to 4-5... Each step of the rough distance information calculation procedure.

Claims (5)

An apparatus for generating a stereo image from a plurality of images taken with or without illumination or intensity of the shooting target,
Image recording means for recording the plurality of images,
Depth value calculation means for calculating a depth value of a shooting target based on a comparison operation of pixel values of corresponding pixels in the plurality of images,
A simple stereo image input device comprising: a stereo image generating means for generating a stereo image based on the depth value obtained by the depth value calculating means. A method of generating a stereo image from a plurality of images taken by changing the presence or absence or intensity of illumination for the shooting target,
An image recording step of recording the plurality of images,
A depth value calculating step of calculating a depth value of a shooting target based on a comparison operation of pixel values of corresponding pixels in the plurality of images,
A stereo image generating step of generating a stereo image based on the depth value obtained by the depth value calculating means. A simple stereo image input method according to claim 2,
A process of photographing a plurality of images in which the conditions of the lighting device have been changed, and dividing the distance information of the photographing target into a plurality of levels by image calculation;
A step of converting the plurality of levels of distance information into distance information actually desired to be displayed,
Converting the distance information into parallax information of a stereo image,
Generating a stereo image based on the parallax information;
Interpolating missing data generated by the generation of the stereo image using surrounding image data. A simple stereo image generating program, wherein the program is a program for causing a computer to execute the steps in the simple stereo image input method according to claim 2. 4. A simple stereo image characterized by being a program for causing a computer to execute the steps in the simple stereo image input method according to claim 2, wherein the program is recorded on a recording medium readable by the computer. A recording medium on which a generation program is recorded.

Images