JP2003209858A - Stereoscopic image generating method and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for generating an easy to see stereoscopic image. <P>SOLUTION: In a virtual view point image generation for generating a stereoscopic image, adjusting the depth or the parallax adjusts the parallax between generated virtual view point images. A particular range in the image is selected by using various segmentation methods and entry using a GUI from a user even in the case of input parallax image resulting from photographing a complicated scene, and adjusting only the parallax or the depth in the particular area eliminates the need for revising the parallax or the depth of pixels in an unexpected area. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method for generating a stereoscopic image of a scene or an object, a method for generating a stereoscopic image that is easy to see, and a stereoscopic photographic printing system provided with this image processing method.

[0002]

2. Description of the Related Art Conventionally, as a method of forming a stereoscopic image, multi-view stereoscopic images such as integral photography (hereinafter referred to as IP), lenticular plate three-dimensional image, and holographic stereogram are particularly known ( Takayoshi Ogoshi: "3D image engineering", Industrial Books, 1972).

In the production of such a multi-view stereoscopic image, images from multiple viewpoints are required, and a large number of pairs of parallax images formed from the multi-view images, that is,
From the parallax image pair, the images of the pair of parallax images are independently observed by the left and right eyes of the observer, thereby enabling stereoscopic viewing. FIG. 2 shows the relationship between the amount of parallax between the parallax images observed by the left and right eyes and the amount of protrusion and depression on the display surface.

In FIG. 3, s10 is the right eye of the observer, s
11 left-eye display surface of the observation distance D, s14 to the viewpoint of the base length K, s13 and s12 from the print surface, and the amount Z _b sinking the pop-out amount Z _f, s16 and s15.

◯ is an image of an object having no parallax, Δ is an object having a parallax amount between parallax images, which is located at a shifted position in each parallax image, and R is an image selectively incident on the right eye s10. , L are images incident on the left eye s11.

Since the point where the image becomes one point by both eyes is in front of the parallax 0 point position of the parallax, the observation distance from the print surface to the viewpoint is set as D, and the parallax amount on the print surface is x _f. , And the protrusion amount s15 is Z _f ,

[0007]

[Equation 1]

[0008] Further, when the image of (L) enters the left eye and the image of (R) enters the right eye, the position where the image becomes one point is behind the print surface, and the amount of parallax on the print surface is x _f. ,, and the subduction amount s16 is Z _b ,

[0009]

[Equation 2]

It can be seen that That is, in 3D printing, the physical pop-out amount due to binocular parallax,
The amount of subsidence and the amount of subduction are simply determined by the amount of displacement (parallax amount) of the subject between the parallax images on the print surface. The relationship between the parallax amount (deviation amount) between the parallax image pairs on the print surface and the parallax value (pixel deviation amount) between the parallax image pairs in the image data depends on the display size of the parallax image on the print surface ( Expansion rate)
, And data-size (number of pixels), observation distance, baseline length, and other geometrical relationships can be easily converted.

Conventionally, a multi-lens camera and a special tripod are required to acquire a multi-viewpoint image. In the method using these multi-lens cameras and a special tripod, the photographing camera group and the position of the subject at the time of photographing Due to the relationship, the three-dimensional effect, that is, the amount of protrusion and the amount of depression is determined. Therefore,
The design of stereoscopic images at the time of shooting was important.

As a guideline for designing a stereoscopic image in capturing a stereoscopic image, the baseline length of the photographing camera is adjusted so that the maximum protrusion amount and the maximum depression amount are within a maximum parallax range. The convergence angle of the camera is adjusted so that the gazing point position (parallax 0 position) of the camera is set on the main subject. It is possible.

First, the maximum parallax range in is the fusion possible range due to the visual function of the human body and the blurring of the image which is felt due to crosstalk when deviating from the optimum viewpoint position.
This is an allowable parallax range determined by an allowable range that does not cause deterioration in the quality of the stereoscopic image that the observer feels to be recognized.

As a value of the fusion possible range, 3D-HDTV
The empirical value for the is suggested in the paper ("Design of 3D video", Noriaki Kumada, Broadcasting Technology 1992.1
1) However, due to the influence of crosstalk when deviating from the optimum viewpoint position, the lenticular print, etc., in which the viewpoint position cannot be limited due to the shape accuracy of the wrench, etc., has a parallax in a narrower range than the range that can be fused. In some cases, it may be better empirically to limit the amount (protrusion amount, subduction amount). Here, the maximum value and the minimum value of this range are the maximum parallax range and the minimum parallax range.

The reason for setting the gazing point position of the camera on the main subject is that if there is a difference between the focus position adjustment position of the eye lens and the depth position of the display image, a feeling of fatigue and difficulty in viewing are induced. Therefore, by reducing the amount of parallax (the amount of protrusion and the amount of depression) of the portion corresponding to the main subject of the image as much as possible, it is possible to empirically generate a stereoscopic image that does not feel tired and that is easy to see.

With the recent development of computer technology,
A method of generating an image from a virtual viewpoint based on a small number of photographed images has been proposed.

One of the methods for generating a virtual viewpoint image based on a photographed image is a model for generating an explicit three-dimensional geometric model of a subject in the virtual viewpoint image generating process.
Based Rendering Method and Image Based Render without Generating a 3D Geometric Model
There is an ering method.

Model Based Renderi
With the ng method, once a three-dimensional geometric model of a subject scene is acquired, it becomes possible to easily observe a subject image from an arbitrary viewpoint by using a computer graphics technique. However, it is very difficult to obtain a complete three-dimensional geometric model of the real world, so it is difficult to generate a virtual viewpoint image with high realism.

On the other hand, Image Based R
Since the ending method is a method that uses a captured two-dimensional image, it is possible to generate a virtual viewpoint image with high realism.

In addition, one method is view mor.
phing (Steven M. Seitz "Vie
w Morphing ”, SIGGRAPH, p21-
p31, 1996) realizes a correct projection geometrical relationship by performing conversion between images so that the line-of-sight directions of images including the same object are perpendicular to the baseline between photographing cameras and the line-of-sight directions are parallel to each other. Therefore, by not using the three-dimensional geometric shape model of the object in the scene explicitly, but by using the pixel correspondence information between the acquired images or the depth information calculated by that, on the straight line connecting the viewpoints of the shooting cameras. This is a method of generating an arbitrary viewpoint image in which a smooth change without distortion can be obtained as if a three-dimensional model exists in the image generation of the arbitrary viewpoint.

Since a realistic multi-viewpoint image can be obtained, there is an advantage that a high-quality stereoscopic image composed of the multi-viewpoint image can be obtained.

In the method of generating an image at a virtual viewpoint based on a small number of photographed images by using these computer technologies, even after capturing an input parallax image, multi-viewpoint images composed of a large number of virtual viewpoint images are displayed. There is an advantage that it is easy to adjust the stereoscopic effect.

For example, Model Based Ren
Since the dering method acquires a three-dimensional geometric model of the subject scene, it is possible to easily observe the subject image from any viewpoint using the technique of computer graphics, as well as the positional relationship of objects in the scene. Can be easily changed, so that it is easy to customize the scene such as moving the individual objects in the scene in the front-rear direction (corresponding to the adjustment of the protrusion amount and the depression amount from the display surface).

However, in reality, it is very difficult to obtain a complete three-dimensional geometric model of the real world.
It is difficult to generate a virtual viewpoint image with high realism. Furthermore, for each object in the scene, we obtain a 3D geometric model of each object, customize the scene, and It is even more difficult to adjust the amount of protrusion and the amount of depression from the display surface.

On the other hand, in the recent method of generating virtual viewpoint images, the method using View Morphing makes it possible to easily observe a subject image from an arbitrary viewpoint on a straight line connecting the viewpoints of the photographing cameras. , It does not explicitly generate 3D geometric models of objects in the scene, and naturally does not have 3D geometric models of individual objects in the scene. It was difficult to adjust the amount of protrusion and the amount of depression from the display surface.

For this reason, a viewpoint corresponding to the conditions of the stereoscopic image design guideline, the maximum pop-out amount and the maximum subsidence amount are within the maximum parallax range, and the viewpoint is effectively used. The parallax of the main subject position in the image is set to 0 as much as possible. It is difficult to satisfy both of these conditions effectively at the same time, and it is difficult for this method to create a stereoscopic image that is easy to see while producing a strong stereoscopic effect.

That is, the multi-view image is used while taking advantage of the View Morphing method capable of acquiring a realistic multi-view image without explicitly acquiring the three-dimensional geometric shape model of the object in the scene. It has been difficult to adjust the stereoscopic effect of the stereoscopic image to a state in which it is easy to see without fatigue.

[0028]

[Problem to be Solved by the Invention] By adjusting the pop-out amount and the sinking amount of a stereoscopic image and making the viewer feel a strong stereoscopic effect at the time of display, the stereoscopic image is less likely to be tired even if the user deviates from the optimum viewpoint position, and the stereoscopic image is less discomforting. To provide a stereoscopic image generation method for easily realizing display.

By adjusting the pop-out amount and the subduction amount of the stereoscopic image, even with respect to the input parallax image in which a complicated scene is photographed, a strong stereoscopic effect is felt at the time of display without distorting an unexpected image region. To provide a stereoscopic image generation method that easily realizes stereoscopic display with less discomfort even if the eyes are out of the optimum viewpoint position.

[0030]

In generating a virtual viewpoint image for generating a stereoscopic image, a parallax between virtual viewpoint images to be generated is adjusted by adjusting a depth amount or a parallax value.

Even for input parallax images of complicated scenes, various segmentation methods and G
Do not change the parallax value or the depth amount of the pixel of the unexpected area by selecting the specific range in the image using the input using the UI and adjusting only the parallax value or the depth amount of the specific region. To

[0032]

BEST MODE FOR CARRYING OUT THE INVENTION (First Embodiment)
On a straight line connecting the viewpoints of the input parallax images captured from one viewpoint, or a virtual viewpoint image at a virtual viewpoint on the extension line, and one of the input images selected as the reference image among the input parallax images, and In the stereoscopic image creating method of acquiring a multi-viewpoint image used for creating a stereoscopic image, which is composed of only the virtual viewpoint image or the virtual viewpoint image, and before performing the virtual viewpoint image generation in the stereoscopic image creating method of combining the stereoscopic images, The stereoscopic effect in the stereoscopic image is presented by adjusting the value of the depth image or the parallax value distribution image, which is an index of the warp amount in the virtual viewpoint image generation obtained from the input parallax image itself or other distance image acquisition means. It is an image generation method using the present invention for adjusting a parallax amount between multi-viewpoint images and generating a stereoscopic image that is easy to observe.

A virtual viewpoint image generation method in this embodiment,
And the stereoscopic image generation method are the image processing device, stereoscopic photo print system, image processing method, virtual viewpoint image generation method and stereoscopic image generation based on a part of a medium recording the stereoscopic printing method and the processing program of Japanese Patent Application No. 4241004. Is the way.

FIG. 1 shows the algorithm of the stereoscopic image generation method of the first embodiment using the present invention. In the figure, s1 is an image data acquisition unit, s2 is a corresponding point extraction unit, s3 is a parallax distribution image creation unit, s4 is an image generation parameter acquisition unit, and s5.
Is a depth feeling adjusting unit, s6 is a virtual viewpoint image sequence generating unit, s7 is a three-dimensional stripe image generating unit, and s6 is a printing unit.

The present invention is an image processing program which operates on a general-purpose PC or a workstation dedicated to image processing, or can be incorporated in hardware having an image processing function.

The operation of the above configuration will be described below.
In the data acquisition unit of s1, the two captured image data are captured. The photographing is performed by two digital cameras mounted substantially parallel to each other along the horizontal direction of the image pickup surface. Alternatively, the two digital cameras may shoot with the main subject placed at the gazing point and with convergence. Alternatively, two cameras may be arbitrarily selected from a camera array including a plurality of digital cameras. Alternatively, a compound eye camera that has two or more optical systems and an imaging system in one housing and is capable of simultaneously capturing images from multiple viewpoints may be used. Alternatively, a stereo adapter, which is an optical system that projects images from two viewpoints in parallel on one imaging surface of the camera by passing them through a prism or mirror in the optical path, is attached to the camera to capture an input parallax image. May be done.

When there is congestion between input data images, distortion correction processing is performed in the data acquisition unit s1. In the distortion correction processing, the trapezoidal distortion that has occurred during the capturing of a stereo image is corrected. By this processing, the viewpoint is converted into a pair of images arranged in parallel in the horizontal direction in the plane perpendicular to the photographing optical axis. Shooting parameters of each camera,
Alternatively, the virtual matrixes after the trapezoidal distortion correction are parallel to each other with the center of the image of each of the left and right image data as the center, using the basic matrix obtained by the general camera calibration method from the captured image itself. Keystone distortion correction.

That is, each image inevitably has a trapezoidal distortion because the left and right optical paths have a predetermined convergence angle. The trapezoidal distortion correction process is a well-known geometric transformation by a three-dimensional rotation matrix of an image.

In this embodiment, the image data is processed as a two-dimensional array of digital data in which each pixel value represents a brightness level.

The corresponding point extraction unit of s2 extracts corresponding points that represent the same subject portion between the input parallax images as point-to-point and pixel-to-pixel correspondence. As a method for extracting the correspondence of each pixel between images, a feature-based method and a region-based method are well-known representative examples.

In the present embodiment, the corresponding point extracting means will be described by using a simple template matching method using the sum of differences, which is one area-based method.

Here, first, template matching is performed with the left image corresponding to the left side of the taken viewpoint position of the two image data (the image corresponding to the right side of the taken viewpoint position as the right image) as the reference image. Then, the corresponding points based on the left image are obtained. First, a partial area of a predetermined size centered on a predetermined point is cut out as a template from the left image. Then, template matching is performed at predetermined pixel intervals. Corresponding points may be obtained for all points in the image, but if the image size is large, the predetermined time will be enormous.

Then, in a predetermined corresponding point search area in the right image, the correlation value between the image area of the same size as the template centering on the pixel of interest in the corresponding point search area and the template of the left image is obtained. .

In the search area for the corresponding points, since a pair of stereoscopic images for parallel viewing has been obtained by the distortion correction processing of step s1, a predetermined horizontal range in the right image is set around the point of interest of the left image. To do.

When the depth preliminary information such as the distance range of the object is given in advance, it is desirable to limit the horizontal search range as much as possible in order to prevent erroneous correspondence of the corresponding point search.

The result of the correlation calculation is obtained as a one-dimensional correlation value distribution having the same size as the predetermined corresponding point search area set in the right image. The corresponding point position is determined by analyzing this one-dimensional correlation value distribution. The corresponding point position is determined to be the position with the smallest sum of differences.

However, when the difference sum at the corresponding point position is larger than a predetermined value, when the difference between the difference sum at the corresponding point position and the second smallest difference sum is smaller than the predetermined value, or near the corresponding point position. If the change in the sum of differences at is smaller than a predetermined value, it is considered that the reliability of the corresponding point extraction processing is low, and therefore that point is not supported.

Regarding the corresponding points for which correspondence has been found, conversely, the template is cut out centering on the corresponding points in the right image,
Similarly, in the left image, set the corresponding point search area,
Perform a horizontal search.

It is judged whether or not this template matching result is in the vicinity of the corresponding point position in the left image. If the result is more than a predetermined position, it is highly likely that it is an incorrect correspondence. Give information. The above calculation is performed for the pixels of the left image at predetermined intervals.

By these processes, the stereo corresponding points are obtained by the template matching based on the left image.

FIG. 4 shows a diagram of corresponding point extraction based on the left image. The left image is the standard image, and the corresponding dots are searched for in the right image, which is the reference image, using the area surrounded by the squares as the template and the dots in the figure as the reference point.

In the parallax value distribution image generation unit of s3, a parallax value distribution image or a depth image is generated. The parallax value for the target pixel can be obtained from the difference in the position on the left and right images of the coordinate value of the corresponding point obtained in s2.

Then, from the parallax values found so far, parallaxes are found also for points (pixels) for which corresponding points were not originally found and for uncorresponding points (pixels) for which corresponding points were not found. A dense parallax value distribution image is obtained by performing interpolation processing using neighboring calculated parallax information. In the present embodiment, as interpolation processing, weighted averaging is performed using the distance parameter at the corresponding point position for which the parallax value has been obtained by the non-interpolated corresponding point and the corresponding point extraction processing as a weight, and the corresponding point has not been obtained originally.
Also, the parallax of uncorresponding uncorresponding points is obtained by the following equation (3).

[0054]

[Equation 3]

N is a parameter that is set arbitrarily to influence the interpolation result, and if the value of n is too small, the obtained parallax becomes close to the average over the entire image, and the parallax distribution becomes uniform. Further, if the value of n is too large, it is greatly affected by the corresponding points of perception. Considering the calculation time, it is desirable that n = 1.

By the above processing, a dense parallax distribution image for the left image is obtained.

When a dense parallax distribution image for the right image is required in the processing of the next step s6, wi and di are replaced to obtain a dense parallax distribution image for the right image as follows.

[0058]

[Equation 4]

As a parallax interpolation method, weighted averaging is performed using a distance parameter as a weight. In addition to the distance, pixel values of RGB channels are added as parameters, and pixels having similar spatial positions and pixel values are used. May be complemented by increasing the weight of.

Further, for example, bilinear interpolation or spline interpolation may be used. Further, when the camera parameter is obtained by the calibration method or the like, it is possible to easily convert the parallax value distribution image between the parallax images into a depth image by using a well-known geometrical relationship. . The parallax value distribution image may be replaced with the depth image, and the process of s4 or less may be performed. Therefore, it is easy to replace the parallax value distribution image of s4 or less with a depth image, the parallax value with a depth value, various parallax reference values, and the threshold with various depth value reference values and thresholds to perform the processing of s4 or less. Can be said to be

In the image generation parameter acquisition unit in s4, the number of images of the virtual image sequence generated in the virtual image sequence generation unit in s6 and the parameter relating to the viewpoint position or the observation position are acquired.

In this embodiment, as parameters relating to the viewpoint position or the observation position, information relating to information about how many images are finally skipped with respect to the image sequence to be observed as a parallax image pair, and the maximum The amount of protrusion and the maximum amount of subduction are required.

FIG. 8 shows the relationship between the observation position and the number of image skips. K is the base line length, D is the observation distance, f is the focal length of the wrench, Δp is the print pitch of one virtual viewpoint image,

[0064]

[Equation 5]

Is the number of image skips. Further, FIG. 9 shows an example of selection of a parallax image pair in an image sequence when one image is skipped. When the observation position becomes closer to the print, the adjacent sequence images are not observed as a parallax image pair, but the images are skipped and the images are observed in the virtual image sequence. The number of image skips can be geometrically determined from the observation distance, the focal length of the wrench, the print pitch for one image, and the base line length.

Further, the viewpoint position of the image generated by the virtual viewpoint image sequence generation unit in s6, which will be described later, is the amount of parallax generated between the parallax image pairs that can be incident on the left and right eyes in the image sequence based on the assumed observation position. , The maximum projecting parallax value and the maximum sinking parallax value are determined so as to fall within a predetermined parallax amount. If the amount of parallax becomes too small, the stereoscopic effect when observing a stereoscopic image will be impaired.
Conversely, if the amount of parallax becomes too large, the stereoscopic image to be observed becomes unclear due to crosstalk with adjacent images.

Therefore, in this parameter acquisition unit, the maximum parallax range, that is, the maximum protrusion amount and the maximum depression amount are set, and the maximum protrusion parallax value between the viewpoints of the virtual viewpoint image and the maximum sinking parallax are set based on the geometrical relationship. Calculate the value,
Set as a reference value between viewpoints.

Since the combination of parallax image pairs selected from the virtual image sequence changes depending on the number of image skips, it is calculated from the maximum pop-out parallax value D' _max and the maximum subduction parallax value D' _min , which is applied in s5. Maximum disparity value D between adjacent viewpoints in a virtual image sequence
_{The max} and the minimum possible disparity value D _min change in the following relationship.

[0069]

[Equation 6]

Further, the viewpoint positions are determined so that the virtual viewpoints are arranged at equal intervals based on the viewpoint position of either the left image or the right image.

This is because the viewpoint positions of the image sequence are arranged at equal intervals to present a natural image change and a stable stereoscopic image with respect to the viewpoint movement, and the stereoscopic effect is adjusted to adjust the input left-right parallax. This is because both images cannot be included in the image sequence.

In the depth sensation adjusting unit at s5, the parallax value distribution image is adjusted so that the parallax value distribution image becomes a legible stereoscopic image. Along with this, the viewpoint position is set using the viewpoint position or observation position parameter set in s4.

The maximum projection amount and the maximum subduction amount are kept within a certain maximum parallax range, and a viewpoint for effective use is taken. Also, the parallax of the main subject position in the image is set to 0 as much as possible. The depth value or the parallax value distribution is adjusted so as to satisfy the above condition.

When the main purpose is to satisfy the above condition, the parallax observed between the left and right images between adjacent viewpoints is the maximum parallax value d.
_{Assuming max} and the minimum parallax value d _min , it is desirable to take d _max ≈D _max and d _min ≈D _min at an arbitrary viewpoint r _x to effectively use the maximum parallax range.

Therefore, offset addition is performed on the entire parallax value distribution image to adjust the parallax value range.
The offset value sh is obtained, the parallax value distribution is adjusted, the maximum depth range defined by the maximum possible parallax value and the minimum possible parallax value is effectively used, and the parallax amount of the main subject is basically 0 as much as possible. The viewpoint position is determined so as to be close to.

First, the parallax values in the parallax value distribution image are subjected to linear adjustment such as offset addition, and then the viewpoint position is determined. The maximum parallax value in the parallax value distribution image is d _max , the minimum parallax value is d _min, and the left image of the input parallax images is the reference image (start point image) of the virtual viewpoint image sequence generated in s7.

Then, the ratio of each image to the parallax between the pair of left and right stereo images, that is, the viewpoint position parameter is r,
If the viewpoint of the left image in the input parallax image is taken, r =
0, when the viewpoint of the right image is taken, r = 1.

Then, by substituting the maximum parallax value and the minimum parallax value into the following equation, the maximum parallax in the x direction generated between the left input image which is the reference image and the virtual start point image when the viewpoint position r is taken. The value and the minimum parallax value can be obtained.

[0079]

[Equation 7]

Although the viewpoint position r is set within a range that does not exceed the maximum subduction amount and the maximum protrusion amount, in order to effectively use the maximum parallax range, the parallax value of the entire parallax distribution image is set to the above-mentioned value. It will be necessary to add an offset. The offset sh and the viewpoint position r adjacent to the reference image are calculated by the following formula.

[0081]

[Equation 8]

It is assumed that

When the main purpose is to satisfy the above, the offset sh is given by assuming that a typical parallax value or depth value of a portion including the main subject is d ₀ .

[0084]

[Equation 9]

It is calculated by

The viewpoint position parameter of the virtual viewpoint image adjacent to the reference image is

[0087]

[Equation 10]

It is obtained by a small r.

However, it is difficult to satisfy both and at the same time and to effectively use the maximum parallax value range only by this processing of adding the offset.

Therefore, the parallax value of the parallax value distribution image is non-linearly converted to adjust the parallax value so that the design condition and the condition of the stereoscopic image are simultaneously satisfied and the maximum parallax value range is effectively used. Change to the depth value configuration of the eye stereoscopic image. FIG. 5 shows a method for adjusting the depth value of an image in a non-linear manner.

FIG. 5 shows a state of processing for an image in which the above-mentioned pointer is prioritized and an offset is added to the entire parallax value distribution image. The horizontal axis direction is the distribution of parallax values before correction, and the vertical axis direction is the parallax value after correction.
Black circles are typical values of parallax values of the main subject.

The dotted line in the center shows the position of parallax 0 in the corrected parallax distribution. The upper dotted line is the position of the maximum possible parallax in the corrected parallax distribution, and the lower dotted line is the position of the minimum possible parallax in the corrected parallax distribution.

In this figure, the representative parallax value of the main subject has a negative value. Therefore, by raising the black circle on the control curve to the position of the dotted line in the figure, the representative value of the corrected main subject can be set to 0, and the above condition can be satisfied.

Further, by smoothly fitting the curve shape in the control curve and smoothing the change in the parallax value, the distortion in the virtual viewpoint image is reduced.

Further, FIG. 6 shows a state of processing for an image in which the above-mentioned pointer is prioritized and an offset is added to the entire parallax value distribution image.

Similar to FIG. 6, the horizontal axis represents the distribution of parallax values or depth values before correction, and the vertical axis represents the parallax value after correction. The black circles are typical values of the parallax values of the main subject, the upper dotted line indicates the position of the maximum possible parallax in the corrected parallax distribution, and the lower dotted line indicates the position of the minimum possible parallax in the corrected parallax distribution. The upper right black circle indicates the maximum parallax value in the parallax value image, and the lower left black circle indicates the minimum parallax value in the parallax value image. Since the maximum parallax value and the minimum parallax value in the parallax distribution image have changed after the main adjustment is performed, the viewpoint position ratio r should be calculated using Equation (6) again.

In this embodiment, the parallax value distribution image or the depth image is adjusted in the order of linear adjustment and non-linear adjustment. However, only linear adjustment, non-linear adjustment, or non-linear adjustment and linear adjustment are performed. You may perform a depth feeling adjustment process in order of adjustment.

In addition to satisfying the policy of,
It is obvious that other effects such as emphasizing a partial stereoscopic effect can be obtained by performing the operation as shown in FIG.

In the virtual image sequence generation unit in s6, the input parallax image, the corrected parallax value distribution image obtained in the depth feeling adjustment unit in s5, and the viewpoint position, that is, the pair of input left and right stereo images corresponding to the viewpoint position. A virtual viewpoint image at each viewpoint position is generated using the ratio r of each image to the parallax.

First, a new viewpoint image from the left image is generated by forward mapping using the pixels of the left image.
That is, first, the parallax d at the pixel position (x, y) in the new viewpoint image that maps each pixel of the left image, the ratio r indicating the viewpoint position, the parallax value for the entire image in the corrected parallax value distribution image. Using the parallax value change amount Δd _L added by the non-linear adjustment of the depth sensation adjusting unit of the offsets sh and s5 of, the following expression (2) is used.

[0101]

[Equation 11]

Similarly, a new viewpoint image from the right image is generated by forward mapping using the pixels of the right image. That is, the position (x _N , y _N ) in the new viewpoint image to which each pixel of the right image is mapped is the parallax d at the pixel position (x, y) of the right image, the ratio r indicating the viewpoint position, and the parallax with respect to the entire image. Disparity value change amount Δ added by the user in the depth sensation adjusting unit for the value offsets sh and s5
Similar to the left image, it is obtained by the following equation (2 ′) using d.

[0103]

[Equation 12]

It is

Then, the pixel at the pixel position (x, y) of the left image is mapped to (x _N , y _N ) of the new viewpoint image from the left image. If pixels at different positions in the input parallax image are mapped to the same pixel due to occlusion due to a change in the viewpoint, and if the parallax values of those pixels are largely different, only the pixel value of the pixel having the large parallax value is adopted. When the parallax values have almost the same value, the average of those pixel values is adopted.

Similarly, the pixel at the pixel position (x, y) of the right image is mapped to (x _N , y _N ) of the new viewpoint image from the right image. If the pixels at different positions in the input parallax image are mapped to the same pixel due to occlusion due to the viewpoint change, as in the case of the left image, and if the parallax values for those pixels are significantly different, the pixel value of the pixel with the smaller parallax value. Adopt only. When the parallax values have almost the same value as in the left image, the average of those pixel values is adopted.

This process is repeated for all pixels of the left and right images.

Then, the new viewpoint image from the left image and the new viewpoint image from the right image are combined. Pixels to which pixels have been assigned only from the left image adopt pixel values from the left image, and pixels to which pixels have only been assigned from the right image adopt pixel values from the right image.

For pixels to which pixel values are assigned from both the left and right images, the pixel value assigned by the left image is adopted when the ratio indicating the viewpoint position is near r = 0 or when r <0. In the vicinity of = 1 or when r> 1, the pixel value assigned by the right image is adopted.

This means that only the pixel value assigned from the input parallax image closer to the viewpoint position of the input parallax image or the extrapolated viewpoint position of the new viewpoint image is adopted. .

Then, the ratio representing the viewpoint position is 0 <r <1.
In the case of, using the viewpoint position as a weight, a weighted average is performed between the pixels assigned from the left and right images by the following equation (3).
Find the pixel value of the composite image.

[0112]

[Equation 13]

N is a predetermined parameter, and if the value of n is too large, the pixel value to be obtained is strongly influenced by the pixel value of the input image in the vicinity. Considering the calculation time, n =
About 1 is desirable.

The pixel filling processing is performed on the pixels to which the pixel is not assigned from either of the left and right images.

In this hole filling processing, an effective pixel which is a predetermined distance away from the pixel to be obtained is searched, and a weighted average value is assigned with the distance of the pixel value as a parameter. When there is no effective pixel, the search range is expanded and the search is repeated.

With the above processing, a new viewpoint image in which all pixels are effective is generated. This process is repeated for the number of viewpoints to form a multi-view image sequence.

The three-dimensional stripe image synthesizing section in s7 synthesizes a three-dimensional stripe image from the multi-viewpoint image sequence.

At this time, the three-dimensional image is combined so that the pixels at the same coordinates of each image in the multi-view image sequence are arranged as adjacent pixels according to the viewpoint arrangement of the images.

In the synthesis, the images of the respective viewpoints are decomposed into strips in the vertical direction line by line, and the images are combined in the reverse order of the viewpoint positions by the number of viewpoints.

Here, the order of arrangement of the viewpoint positions is set in the reverse order when the lenticular plate is observed.
This is because the image is observed left-right in the pitch.

The printing section s8 prints a three-dimensional stripe image. Before printing, s1 to s
The processing results up to 7 may be displayed on the display device so that they can be confirmed.

A good three-dimensional image can be observed by superimposing a lenticular plate on the image printed by the processing of s1 to s8.

In this embodiment, the camera arrangement is limited to the horizontal direction, and the processing method is described only when the input parallax image pair has the parallax only in the horizontal direction. However, the camera arrangement is set to the vertical direction. It can be said that the virtual viewpoint image sequence can be easily created by applying the processing method by replacing the arrangement from the horizontal direction to the vertical direction with respect to the input parallax image pair photographed as described above. Therefore, it can be said that the process can be easily extended to the creation of an IP print having horizontal and vertical parallax.

[0124] Further, similarly to the multi-view stereoscopic print using the lenticular sheet shown in this embodiment, the IP print, the lenticular, the multiview stereoscopic display using the IP, the holographic stereogram, and the like, the viewpoints are changed little by little. The multi-viewpoint image sequence is generated by the image generation method according to the present embodiment, and the multi-viewpoint image sequence is used as an input image for each of the multi-viewpoint stereoscopic display methods that require a large number of captured images. By a combining method suitable for the stereoscopic display method, it is combined as a multi-view stereoscopic image,
Even if it is displayed and output, the parallax between the multi-viewpoint images is adjusted, and the protrusion amount and the depression amount of the object in the image in the stereoscopic image are adjusted. By using the elements of the present invention, an easily viewable stereoscopic image is generated. can do.

This method has an advantage that the number of cameras for photographing multi-viewpoint images is conventionally small. Then, particularly by using the element of the present invention as shown in s5, the parallax between the multi-viewpoint images is adjusted, and the protrusion amount and the depression amount of the object in the image in the stereoscopic image are adjusted, so that it is easy to see. It is possible to generate a stereoscopic image.

(Second Embodiment) In the present embodiment, the method of the first embodiment is applied to an input parallax image obtained by photographing a complicated scene, and an unexpected area which is generated when depth adjustment processing is performed is performed. The problem that the parallax value or the depth amount of the pixel is changed and the image of the unexpected image region is distorted is prevented.

Therefore, area division is performed using the information of the input stereo image, the depth image, the parallax value distribution image, or a plurality of these images, and the area information obtained by the area division is used to select a specific area and specify it. Only the depth amount or parallax value of the pixels corresponding to the region is adjusted.

Thus, the depth adjustment is performed without changing the depth amount or the parallax value of the pixels in the unexpected area, so that the image of the unexpected image area is not distorted and the virtual viewpoint images are generated. This is a method of adjusting parallax to generate a stereoscopic image that is easy to see.

FIG. 2 shows the algorithm of this embodiment. In the figure, s201 is an image data acquisition unit, s202 is a corresponding point extraction unit, s203 is a parallax distribution image creation unit, s204 is an image generation parameter acquisition unit, s205 is a region division unit, s
Reference numeral 206 is a depth feeling adjusting unit, s207 is a virtual viewpoint image sequence generating unit, s208 is a three-dimensional stripe image generating unit, and s209 is a printing unit.

In the present embodiment, the image data acquisition unit of s201, the corresponding point extraction unit of s202, the parallax distribution image creation unit of s203, the image generation parameter acquisition unit of s204, and the virtual viewpoint image sequence generation unit of s207, s208.
The three-dimensional stripe image generation unit of, and the printing unit of s209 are
The image data acquisition unit of s1 of Example 1, the corresponding point extraction unit of s2, the parallax distribution image creation unit of s3, the image generation parameter acquisition unit of s4, the virtual viewpoint image sequence generation unit of s6, s
The same operations as the three-dimensional stripe image generation unit 7 and the printing unit s8 are performed. Therefore, the description is omitted in this embodiment.

In the area dividing unit in s205, the area is divided using the input stereo image, the depth image, the parallax value distribution image, or the composite information of these images, and the specific area is selected.

Regarding the area division processing, known methods include clustering methods such as the k-mean method, processing methods using multi-step thresholds, and area growth methods (Image Analysis Handbook, Mikio Takagi, Yoshihisa Shimoda, The University of Tokyo). Publication (1991)), for application to the present invention,
There is no limitation on the difference between the methods as long as the above-mentioned image information is effectively used and the boundary in the image corresponding to the object contour of each object in the scene is not eroded. Any area division method may be selected.

In the description of this embodiment, as a simple example of the area dividing method, the result that would be obtained when the area growing method is used is used.

FIG. 10 shows an input parallax image. FIG. 11 shows a label image generated as a result of performing area division on the input image in the area division unit in s205.

In the depth feeling adjusting unit of s206, first, similarly to the first embodiment, linear adjustment such as offset addition is performed on the entire parallax value distribution image.

The maximum parallax value in the parallax value distribution image is d _max , the minimum parallax value is d _min, and the left image of the input parallax images is the reference image (start point image) of the virtual viewpoint image sequence generated in s7. . Then, the ratio of each image to the parallax between the left and right stereo image pairs, that is, the viewpoint position parameter is r, and r = 0 when the viewpoint of the left image is taken from the input parallax images, and r when the viewpoint of the right image is taken. = 1. Then, by substituting the maximum parallax value and the minimum parallax value into the following equation, the maximum parallax value and the minimum parallax value in the x direction generated between the left input image which is the reference image and the virtual start point image when the viewpoint position r is taken. The parallax value can be obtained.

[0137]

[Equation 14]

Although the viewpoint position r is set within a range that does not exceed the maximum subduction amount and the maximum protrusion amount, in order to effectively use the maximum parallax range, the parallax value of the entire parallax distribution image is set to the above-mentioned value. It will be necessary to add an offset.

The offset sh and the viewpoint position r adjacent to the reference image are calculated by the following equation.

[0140]

[Equation 15]

It is assumed that

Further, when the main purpose is to satisfy the above, the offset sh is given by assuming that a typical parallax value or depth value of a portion including the main subject is d ₀ .

[0143]

[Equation 16]

It is calculated by The viewpoint position parameter of the virtual viewpoint image adjacent to the reference image is

[0145]

[Equation 17]

It is obtained by a smaller r among them.

However, it is difficult to satisfy both and at the same time and to effectively use the maximum parallax value range only by this processing of adding the offset.

Therefore, in the first embodiment, the parallax value of the stereoscopic image is adjusted by nonlinearly converting the parallax value of the parallax value distribution image in the general manner of performing the tone conversion of the image. The processing is performed to change the depth value configuration of the multi-view stereoscopic image that satisfies the design conditions and at the same time and effectively uses the maximum parallax value range.

However, the parallax value distribution image obtained from the stereo images having the object arrangement configuration as shown in FIG. 10 is subjected to the non-linear adjustment processing of the depth sensation adjusting unit of s5 of the first embodiment. In such a case, the parallax value of the pixel in the unexpected area is changed, which causes a problem of distorting the unexpected image area. This is because the stereo image pair of FIG. 10 has a parallax value distribution as shown in the graph of (1) of FIG.

In FIG. 12, A, B, C and D are s20.
This is a symbol of a region label arbitrarily attached to a region excluding the background region from the region division information obtained by the region division unit 5 and corresponds to A, B, C and D in FIG. The vertical axis of the graph of FIG. 12 shows the distribution of the parallax values in the parallax distribution image, like the vertical axis of the graph of FIG. Label A, respectively
The rectangles in B, C, and D are the parallax value distribution ranges in the respective regions in the parallax image. The black circles in the rectangle are representative values arbitrarily selected in the divided area.

When the object A in FIG. 11 is the main subject, as can be seen from the graph of FIG. 12, the parallax value of the representative point of the main subject is the same as in the adjustment of FIG. 5 of the first embodiment. Since it has a negative value, the parallax value of the representative point and the vicinity thereof will be brought close to 0 as in the adjustment of FIG. 5 of the first embodiment.

However, since the object having the label D in this stereo image also has pixels of the parallax value near the representative point in the main subject A, the non-linearity of the depth feeling adjusting unit of s5 of the first embodiment is used. If such a large adjustment process is performed, the parallax value of the pixels of the area included in the object D where the effect of the parallax adjustment is not expected is changed.

Therefore, depending on the nature of the input parallax image,
When an attempt is made to perform non-linear adjustment processing of the depth sensation adjusting unit of s5 of the first embodiment in the virtual viewpoint image, that is, the stereoscopic image, the parallax value of the pixel in the area where the effect of the adjustment is not expected is changed. Obviously, there may be a problem in that an unexpected image region is distorted.

Therefore, in the second embodiment of the present invention, in contrast to the first embodiment, when the nonlinear depth amount or parallax value is adjusted, area division is implemented and a specific area is selected. Then, by adjusting the depth amount or parallax value of the pixels corresponding to the specific region, the depth adjustment can be performed without changing the depth amount or parallax value of the pixels in the unexpected region, and Adjust the parallax between the generated virtual viewpoint images without distortion.
Generates a stereoscopic image that is easy to see.

In the nonlinear adjustment processing of the depth sensation adjusting unit in s206 of the second embodiment, the depth amount or parallax of the pixel corresponding to the specific area is based on the area information as shown in FIG. 11 generated in s205. Make the desired adjustments to the values.
In the present embodiment, as an example, a method of organizing the parallax values in the parallax distribution image in a graph as shown in FIG. 12 and interactively adjusting the non-linear parallax is shown.

In FIG. 12, A, B, C and D are s20.
This is a symbol of a region label arbitrarily attached to a region excluding the background region from the region division information obtained by the region division unit 5 and corresponds to A, B, C and D in FIG. The vertical axis of the graph of FIG. 12 shows the distribution of the parallax values in the parallax distribution image, like the vertical axis of the graph of FIG. Label A, respectively
The rectangles in B, C, and D are the parallax value distribution ranges in the respective regions in the parallax image. The black circles in the rectangle are representative values arbitrarily selected in the divided area.

In (1) of FIG. 12, the short form of the label B touches the dotted line representing D _max , the short form of the label D touches the dotted line representing D _min, and there is no object having a parallax value protruding from the dotted line. . This fact is nothing but that the policy of stereoscopic image design is established by the linear parallax adjustment in the previous stage.

Therefore, as shown in (2) of FIG. 12, a process of bringing the representative point in the main subject closer to 0 is performed. However, the information obtained by the area division of the second embodiment is used to correspond to the specific area. By only adjusting the depth amount or parallax value of the pixels to be used, without distorting the unexpected image area,
At the same time, it becomes possible to satisfy the policy of stereoscopic image design.

The distribution range of the parallax values in the individual objects, which are shown in the short form, are preserved as shown in (2) of FIG. 12 even if the processing is performed. When it is desired to maintain the accuracy, the parallax value is converted into the depth value and the processing is performed.

In the processing from s207 onward, the same processing as that of the first embodiment is performed to adjust the depth amount or the parallax value of the pixel corresponding to the specific region, thereby the depth amount of the pixel in the unexpected region or By performing the depth adjustment without changing the parallax value, it is possible to adjust the parallax between the generated virtual viewpoint images and generate a stereoscopic image that is easy to see.

This method has an advantage that the number of cameras for taking multi-viewpoint images can be reduced conventionally. Then, particularly by using the elements of the present invention as shown in s205 and s206, by selecting a specific area and adjusting the depth amount or the parallax value of the pixel corresponding to the specific area, By adjusting the depth without changing the depth amount of pixels or the parallax value, without distorting the unexpected image area,
This is a method that can adjust the parallax between the generated virtual viewpoint images to generate a stereoscopic image that is easy to see.

(Third Embodiment) In the present embodiment, in the method for generating an easy-to-see stereoscopic image described in the second embodiment, instead of performing area division, the user interactively divides the area from the GUI. Is the way to do.

FIG. 2 shows the algorithms of this embodiment and the second embodiment. In other words, this embodiment takes the same processing algorithm as the second embodiment. In the figure, s201
Is an image data acquisition unit, s202 is a corresponding point extraction unit, s20
3 is a parallax distribution image creation unit, s204 is an image generation parameter acquisition unit, s205 is a region division unit, s206 is a depth feeling adjustment unit, s207 is a virtual viewpoint image sequence generation unit, s
Reference numeral 208 is a three-dimensional stripe image generation unit, and s209 is a printing unit.

In the present embodiment, the image data acquisition unit at s201, the corresponding point extraction unit at s202, the parallax distribution image creation unit at s203, the image generation parameter acquisition unit at s204, the depth feeling adjustment unit at s206, and the virtual viewpoint image at s207. The sequence generation unit, the three-dimensional stripe image generation unit of s208, the printing unit of s209 are the image data acquisition unit of s201, the corresponding point extraction unit of s202, and s203 of the second embodiment.
The parallax distribution image generation unit of s204, the image generation parameter acquisition unit of s204, the depth feeling adjustment unit of s206, the virtual viewpoint image sequence generation unit of s207, the three-dimensional stripe image generation unit of s208, and the printing unit of s209 perform the same operation. . Therefore, the description is omitted in this embodiment.

In the area dividing unit of s205, the user interactively and highly accurately inputs information about the area division, and means for immediately generating the area division information using the information about the area division. Equipped with.

For example, as shown in Japanese Unexamined Patent Publication No. 200-339483, an edge trace between the start point or the previous fixed point determined by the pointing device and the point currently pointed by the mouse is moused on the image displayed on the display. High-performance that eliminates the need for correction work and enables both high operability and high-accuracy edge tracing to be achieved by constantly performing high-speed tracking of the movement of the mouse and displaying the results on the display in real time. It is provided with a means for generating real-time and interactive area segmentation information. Based on the edge trace result, the area division information used in s206 is immediately generated.

As a result, in the area dividing unit of s205 of the second embodiment, various area dividing methods are executed, and good area dividing is performed such that the input image is divided for each object included in the scene. It is not necessary to set the area division parameter that depends on various area division methods, which is necessary for obtaining the result and changes the division result largely due to a subtle error in the setting value.

In the processing from s206 onward, the same processing as that of the second embodiment is performed to adjust the depth amount or the parallax value of the pixel corresponding to the specific region, and By performing the depth adjustment without changing the parallax value, it is possible to adjust the parallax between the generated virtual viewpoint images and generate a stereoscopic image that is easy to see.

This method has an advantage that the number of cameras for photographing multi-viewpoint images is conventionally small. Then, particularly by using the elements of the present invention as shown in s205 and s206, in the area division, without setting complicated area division parameters,
Appropriate segmentation is possible by user input,
By selecting a specific area from the divided areas and adjusting the depth amount or parallax value of the pixel corresponding to the specific area, the depth of the pixel or the parallax value of the unexpected area can be adjusted without changing the depth. By performing the adjustment, the parallax between the virtual viewpoint images to be generated can be adjusted without distorting an unexpected image region, and a stereoscopic image that is easy to see can be generated.

[0170]

EFFECTS OF THE INVENTION By adjusting the pop-out amount and the sinking amount of a stereoscopic image, it is possible to easily realize a stereoscopic display with less discomfort, which makes the eyes less tired even when the optimal viewpoint position is deviated while giving a strong stereoscopic effect at the time of display. To provide a stereoscopic image generation method to be realized.

By adjusting the pop-out amount and the sinking amount of the stereoscopic image, even for an input parallax image of a complicated scene, an undesired image area is not distorted, and a strong stereoscopic effect is felt at the time of display. To provide a stereoscopic image generation method that easily realizes stereoscopic display with less discomfort even if the eyes are out of the optimum viewpoint position.

[Brief description of drawings]

FIG. 1 is a diagram showing an algorithm according to a first embodiment of the present invention.

FIG. 2 is a diagram showing algorithms of a second embodiment and a third embodiment of the present invention.

FIG. 3 is a diagram showing the relationship between the amount of protrusion and the amount of depression and the amount of parallax.

FIG. 4 is a diagram showing template matching.

FIG. 5 is a diagram showing a feeling of depth adjustment.

FIG. 6 is a diagram showing a feeling of depth adjustment.

FIG. 7 is a diagram showing a feeling of depth adjustment.

FIG. 8 is a diagram showing a relationship between an observation position and image skipping.

FIG. 9 is a diagram showing a relationship of image skipping in an image sequence.

FIG. 10 is a diagram showing a stereo image pair.

FIG. 11 is a diagram showing a result of performing region division on the stereo image of FIG.

FIG. 12 is a diagram showing depth feeling adjustment using area division information.

Claims (6)

[Claims] 1. A stereoscopic image generation method using virtual viewpoint image generation, which comprises adjusting a parallax between virtual viewpoint images to be generated by adjusting a depth amount or a parallax value. . 2. The depth amount or the parallax value is adjusted by performing area division using information of an input stereo image, a depth image, a parallax value distribution image, or a plurality of these images, and selecting a selected specific area. The stereoscopic image generation method according to claim 1, wherein the stereoscopic image generation method is performed. 3. The depth amount or the parallax value is adjusted by performing an area division of an input image by an interactive operation of a user,
The stereoscopic image generation method according to claim 1, wherein the stereoscopic image generation method is performed on the selected specific area. 4. A recording medium on which the stereoscopic image generation method according to claim 1 is recorded as a program. 5. A recording medium on which the stereoscopic image generation method according to claim 2 is recorded as a program. 6. A recording medium on which the stereoscopic image generation method according to claim 3 is recorded as a program.