JPH09179998A - Three-dimensional image display system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain perception to depth even with a single eye with images, such as stereoscopic paired images, which give a stereoscopic effect by being viewed with both the eyes. SOLUTION: An image input part 1 is equipped with a left-eye camera 11 and a right-eye camera 12 and images which give a stereoscopic effect through both the eyes like stereoscopic paired images are inputted. Then a depth information calculation part 2 finds parallax from the images by a parallax calculation part 21 and calculates depth information by a depth calculation part 22 from the parallax. Then a focus effect image generation part 4 generates an image introducing an effect of focus on the basis of the said depth information according to the specification parameters of depth for focusing and a range of focusing inputted by a user through a camera parameter input part 3 and displays the image interactively at an image display part 5. Here, the focus position, etc., is continuously varied from the camera parameter input part 3 to obtain a feeling of depth even when the image is viewed with a single eye.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention interactively displays an image in which the effect of the focus is incorporated, while the user controls the depth and range of the focus from the image in which the sense of depth can be obtained by viewing with both eyes. By doing so, the present invention relates to an image display system capable of perceiving a sense of depth even with a single eye.

[0002]

2. Description of the Related Art Conventionally, there has been known a method for enabling stereoscopic viewing with a sense of depth by viewing two images having parallax (hereinafter referred to as stereo pair images) separately by the left and right eyes. ing.

On the other hand, as a means for obtaining depth information from a plurality of photographs, a method used for obtaining contour lines from a plurality of aerial photographs taken from a certain route is known. This method geometrically obtains the distance to the object by the principle of triangulation, but depth information cannot be obtained from the stereo pair image unless such a method is used.

In order to display an image having three-dimensional information, the user interactively performs an operation such as rotation or movement (moving fast on the front side and slowly moving on the back side) on an object in the image. , There is a method for obtaining a three-dimensional perception even with a single eye.

[0005]

The above-mentioned conventional three-dimensional image display method is mainly used for visualizing a relatively small object such as computer graphics which already has three-dimensional information. It was difficult to apply it to natural images such as landscapes and people photographed by the stereo camera, which do not have three-dimensional information in advance.

An object of the present invention is to display an image, such as a stereo pair image, in which the user can interactively control the focus and the like, based on an image that can obtain a stereoscopic effect using both eyes.
An object of the present invention is to provide a three-dimensional image display system capable of obtaining a perception of depth even with a single eye.

[0007]

In order to achieve the above object, the present invention provides an image input means for inputting an image enabling binocular stereoscopic vision, and a left eye and a right eye obtained by the image input means. Means for obtaining parallax from image signals, means for obtaining depth based on the parallax, camera parameter input means capable of continuously changing camera parameters, and based on the input camera parameters and the depth A three-dimensional image display system is provided which has a means for creating an image having a focus effect and a display means for outputting the created image having a focus effect.

In the above three-dimensional image display system,
It is preferable to obtain the parallax by obtaining the cross-correlation of the left-eye and right-eye image signals, as the means for obtaining the parallax, because it is possible to obtain the parallax easily and accurately.

Further, according to the present invention, a camera parameter input means capable of continuously changing a camera parameter for an image signal having depth information for each pixel, and the depth according to the input camera parameter. A three-dimensional image display system having a means for creating an image having a focus effect based on information and a display means for outputting the created image having a focus effect. Use it as a means.

In the above-mentioned three-dimensional image display system, means for creating an image having a focus effect are means for obtaining a point spread function based on input camera parameters and depth information, and the point spread function and the image. It is preferable that it is configured with a means for performing a convolution calculation of a signal, because an accurate focus effect image can be created.

Alternatively, the means for creating an image having a focus effect is configured by subjecting the image signal to filtering processing based on the input camera parameters and depth information. It is preferable in that it can be done.

According to the present invention, an image such as a stereo pair image which enables a stereoscopic effect to be obtained with both eyes is input from a camera or a scanner, depth information is calculated from the input image, and the focus from the user is determined. Depending on the camera parameters such as the depth of focus and the range of focus, etc., the depth and focus of focus can be adjusted by interactively creating and displaying an image that incorporates the focus effect based on the depth information. By continuously changing the matching range, it is possible to obtain a sense of depth even with a single eye, and it is possible for a person with one eye disability to obtain a sense of depth.

[0013]

Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of the first embodiment of the present invention, and shows an outline in the case of targeting two stereo pair images for the left eye and the right eye.

In the image input unit 1, left-eye and right-eye images of stereo pair images are input separately. Here, the input may be conversion from a photograph, which is analog data, into digital data by using an input device such as a scanner, and may be input by using a digital camera or the like, and the subject may be directly input as digital data. FIG. 1 shows the latter example, which includes a left-eye camera 11 and a right-eye camera 12 and inputs each captured image to the image input processing unit 13 to obtain a stereo pair image.

The depth information calculation unit 2 includes a parallax calculation unit 21 and a depth calculation unit 22, and calculates depth information for each pixel based on the image input by the image input unit 1. The depth information is the distance from the camera lens to the calculation target pixel. There are several possible depth information calculation methods, which will be described later.

In the camera parameter input section 3, the user inputs camera parameters using an input device such as a keyboard and a mouse. Here, the camera parameters are a distance to a focus position (herein referred to as a focus position) and a focus range spreading before and after the focus position, that is, a depth of field. Regarding the distance to the in-focus position, there are cases where it is directly input, and there is a case where the position (pixel) to be focused on the image is designated with a mouse or the like and the depth of the pixel is taken as the distance to the in-focus position. . The detailed input method will be described later.

The focus effect image creation unit 4 includes a point distribution function calculation unit 41 and a convolution calculation processing unit 42, and both the distance to the focus position input by the camera parameter input unit 3 and the depth of field or Based on one side and the depth information calculated by the depth information calculation unit 2, one image simulating the effect of the lens is created. By this processing, an image having a focusing effect in which a focused portion and a non-focused portion are mixed in one image is created. Details of this creating method will be described later.

In the image display section 5, the focus effect image creating section 4
The image created by is displayed on the display. The user views the displayed image, returns to the camera parameter input unit 3 again, and inputs different camera parameters to display images with different focus effects. By repeating this operation and continuously viewing images with different focus effects, the user can obtain a sense of depth.

Next, details of the depth information calculation unit 2 in FIG. 1 will be described. As described above, the depth information calculation unit 2
Is divided into a parallax calculation unit 21 and a depth calculation unit 22.

First, the parallax calculator 21 will be described.
Parallax refers to a shift in pixel position with respect to the same object in each of the left and right images. FIG. 2 shows a situation in which a stereo pair image is photographed from directly above. Hereinafter, in order to simplify the explanation, it is assumed that the parallax is considered only in the horizontal direction, that is, the x direction, and not in the vertical direction, that is, the direction. Since the human eyes are aligned horizontally, this condition is fully realistic.

In FIG. 2, it is assumed that the point O on the subject 6 is imaged on the left-eye image f _l through the lens 71 at a position x _l and on the right-eye image f _r through the lens 72 at x _r . This imaged position is called a corresponding point. in this case,
Parallax dx is the difference between the positions of the corresponding points, i.e., it is defined as dx = x _l -x _r.

In order to obtain the parallax according to the above definition formula, it is necessary to extract corresponding points on the left and right images, and as an example thereof, the normalized cross correlation (Normali) will be described below.
An example according to the zed Cross Correlation) method will be shown.

If the left-eye image (image captured by the left-eye camera) is f _l and the right-eye image (image captured by the right-eye camera) is f _r , the normalized cross-correlation value φ _NCC is given by the following equation. Is defined in.

[0025]

[Equation 1]

[0026] Here, M _l, M _r: each f _l, the average value sigma _l of f _r, sigma _r: each f _l, f the standard deviation of the _r N: number of pixels i, j: respectively, horizontal and vertical Is a subscript indicating the pixel of
The axis is the direction connecting both eyes, and the j axis is the direction perpendicular to the i axis and coinciding with the direction of gravity.

The summation by Σ is performed up to the upper limit value of i, j.

The operation image of the above equation is shown in FIG. First,
An image of an arbitrary size centered on the pixel position i, j (this is called a search block (211 in FIG. 3)) is extracted from the left-eye image f _l shown in FIG. 3A. Next, FIG.
(B) on the right-eye image f _r as shown in the pixel position i + h, j
A region of the same size as the search block 211 centering around + k is taken out, and φ _NCC is calculated according to the definition formula. This operation is repeated within an arbitrary range (this is called a search range (212 in FIG. 3)) while changing h and k. The pixel position i + h, j + k at which φ _NCC is maximum in the search range 212 is the corresponding point to the pixel position i, j on the left-eye image f _l , and as described above, when only the x direction is considered, h is the parallax dx. Becomes The above operation is repeated for all pixel positions i, j of the left-eye image f _l .

Although the corresponding point calculation method using the normalized cross-correlation method has been described above, the Dot Product effective for an image containing a high spatial frequency component is also available.
Phase Correlation using tCorrelation method or Fourier transform capable of high-speed processing
There are methods such as the law. In the present invention, any method other than the above may be used as long as the corresponding points can be extracted.

Subsequently, the depth calculation unit 22 in FIG.
Calculates the depth of each pixel based on the parallax calculated by the parallax calculation unit 21. The depth z is represented by the following equation using the distance b between the left and right lenses 71 and 72 in FIG. 2 and the focal length f of the lens, and is calculated for all (i, j).

[0031]

[Equation 2]

The above formula is an ideal case where the photographing conditions such as the optical axes of the lenses 71 and 72 are parallel, and it is difficult to satisfy this condition in reality, and it is necessary to perform correction processing. Become. It is safe to assume that these conditions are met, i.e. the above equations are applicable, in the present invention, which is primarily intended for entertainment applications, rather than requiring rigorous measurements such as photographic measurements. . If the parameters b and f in the above equation are unknown, b is the average distance between human eyes,
f may be 50 mm which is a standard lens of a general camera. However, these can also be changed in the system as parameters.

FIG. 4 is a diagram showing an interface configuration example of the camera parameter input unit 3 and the image display unit 5 in FIG. In the image display area 51 on the screen of the image display unit 5, an image with a focus effect is displayed. Initial screens for which no parameters have been entered are displayed with appropriate default values (eg, full area in focus).
At the bottom of the image display area 51, there is a pointer (in this example, the mouse pointer 5) using a pointing device such as a mouse.
There is a slider that can be controlled by 2). The slider changes the parameters continuously. The slider has a portion responsible for two functions, and one is a focus position control slider 54 responsible for the control function of the focus position 53.
It is. The other one is a depth-of-field control slider 55 which controls the depth of field that spreads before and after the focus position.
It is. By moving the in-focus position 53 to the left or right with the mouse pointer 52 or clicking a triangle on the left and right of the in-focus position control slider 54, the in-focus position is adjusted from near to far or from far to near. Also, by clicking the triangles on the left and right of the depth of field control slider 55 that indicates the depth of field, it is possible to widen or narrow the depth of field. As for the in-focus position, by clicking an arbitrary position in the image displayed in the image display area 51 with the mouse pointer 52 or the like, the depth having the pixel at the position is set as the in-focus position. It is also possible. Each time a camera parameter is interactively input by the sliders 54, 55 and the mouse pointer 52, a focus effect image described later is displayed in the image display area 51.

Next, the focus effect image forming section 4 in FIG.
Will be described. As described above, the focus effect image creation unit 4 is divided into the point spread function calculation unit 41 and the convolution calculation processing unit 42.

FIG. 5 is a diagram showing how a point object at the focus position and point objects before and after the focus position form an image. As can be seen from the figure, the point object B at the in-focus position is imaged as a point object on the imaging plane, but the point object A at a position closer to the in-focus position and the point object C at a far position are the imaging surface. At, a blurred image with a certain distribution is obtained. This "blur condition"
Is the point spread function (Point Spread).
Function (PSF) and has the distance between the lens and the object as a parameter. Since the PSF is a function indicating how a point object is imaged, the convolution operation between the PSF and the object may be performed in order to obtain an image of the object having a spread.

In the focus effect image creating section 4 in FIG.
In the point spread function calculating unit 41, the depth, the focus position and the depth of field parameters specified by the camera parameter input unit 3 which have already been obtained by the depth information calculating unit 2 are substituted into the PSF, and the PSF is calculated for each pixel. Is determined, the convolution operation processing unit 42 determines the PSF determined above.
And the actual image (f _l or f _r ) input by the image input unit 1
By performing a convolution calculation with and, an image having a focus effect simulating "blur" is created. The convolution operation is P
The same result can be obtained by performing Fourier transform on each of the SF and the actual image, and then performing the inverse Fourier transform after multiplication in the Fourier space. P of the lens that shot the input stereo pair image
If the SF is known, it may be used, but if it is not known, an appropriate assumption may be made such as approximation with a Gaussian function. Alternatively, when the simulation is not required faithfully optically, the filtering process may be performed by a low-pass filter or the like that appropriately blurs the deviation from the focus position as a parameter without using the PSF.

Next, a second embodiment of the present invention will be described. The present embodiment is intended for an image such as computer graphics to which depth information is given in advance when an image signal is created. Therefore, it is not necessary to calculate depth information from a given image signal,
The image input unit 1 and the depth information calculation unit 2 in the configuration of FIG. 1 for explaining the first embodiment of the present invention described above are unnecessary, and the camera parameter input unit 3, the focus effect image creation unit 4, and the The image display unit 5 corresponds to the configuration of the second embodiment. In the present embodiment, an image signal from computer graphics or the like is used instead of the image signal from the image input unit, and the convolution operation processing unit 42 of the focus effect image creation unit 4 is used.
Instead of the depth information from the depth information calculation unit, the depth information regarding the image signal from computer graphics or the like may be input to the point spread function calculation unit 41.

[0038]

As described above, according to the present invention, after inputting an image having depth information or an image for stereoscopic vision such as a stereo pair image, the position to be focused and the depth of field are set by the user. It is possible to control the image interactively and continuously display images incorporating the focus effect, so it is possible to obtain a sense of depth even with a single eye, and it is possible for people with one eye disability to perceive depth. There is an effect.

[Brief description of the drawings]

FIG. 1 is a diagram showing a first embodiment of the present invention, and is a block diagram showing an example applied when a stereo pair image is input.

FIG. 2 is a diagram showing a situation in which a stereo pair image is photographed in the above embodiment.

3A and 3B are diagrams showing an outline of an example of obtaining parallax using a normalized cross-correlation method in the above-described embodiment.

FIG. 4 is a diagram illustrating an interface configuration example of a camera parameter input unit and an image display unit in the above-described embodiment.

FIG. 5 is a diagram showing how an in-focus position and point objects before and after the in-focus position are imaged in the embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Image input part 11 ... Left eye camera 12 ... Right eye camera 13 ... Image input processing part 2 ... Depth information calculation part 21 ... Parallax calculation part 22 ... Depth calculation part 3 ... Camera parameter input part 4 ... Focus effect image creation part 41 ... Point spread function calculation unit 42 ... Convolution calculation processing unit 5 ... Image display unit 51 ... Image display area 52 ... Mouse pointer 54 ... Focus position control slider 55 ... Depth of field control slider

Claims (6)

[Claims] 1. An image input unit for inputting an image enabling binocular stereoscopic vision, a unit for obtaining a parallax from image signals for the left and right eyes obtained by the image input unit, and a depth based on the parallax. Determining means, camera parameter input means capable of continuously changing camera parameters, means for creating an image having a focus effect based on the input camera parameters and the depth, and the created focus. A three-dimensional image display system comprising: a display unit that outputs an image having an effect. 2. The three-dimensional image display system according to claim 1, wherein the means for obtaining the parallax obtains the parallax by taking a cross-correlation of the left-eye and right-eye image signals. 3. A camera parameter input unit capable of continuously changing camera parameters for an image signal having depth information for each pixel, and based on the depth information according to the input camera parameters. A three-dimensional image display system comprising: a unit that creates an image having a focus effect; and a display unit that outputs the created image having a focus effect. 4. The camera parameter according to claim 1, 2 or 3, wherein either one of a focus position and a focus range or both of them are included. Dimensional image display system. 5. A means for creating an image having a focus effect is means for obtaining a point spread function based on input camera parameters and depth information, and means for performing a convolution operation of the point spread function and an image signal. The three-dimensional image display system according to any one of claims 1 to 4, further comprising: 6. The method according to claim 1, wherein the means for creating an image having a focus effect performs a filtering process on the image signal based on the input camera parameter and depth information. The three-dimensional image display system according to any one of 4 to 4.