JP2011141710A - Device, method and program for estimating depth - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a depth estimating device which can reduce incorrect estimation of a depth value when occlusion occurs among a plurality of camera images and corresponding points do not exist, or even when a plurality of corresponding points exist in a uniform object. <P>SOLUTION: The depth estimating device 1 includes: an image input means 10 which inputs a reference image and a plurality of adjacent images; a corresponding pixel difference operation means 20 which calculates for each assumed depth value, an adjacent image pixel difference value which is a difference absolute value between a pixel value of the pixel of the reference image and the pixel value of the pixel of the adjacent image corresponding to a parallax of an assumed depth value of the pixel; a difference value determining means 30 which determines an average value or a minimum value of the adjacent image pixel difference value as a selection difference value by a threshold; a storage means 40 which associates the selection difference value with the assumed depth value and a pixel position of the reference image to stores it; and a depth value determining means 60 which determines the depth value for each pixel position by searching an assumed depth value where the selection difference value becomes minimum in the storage means 40. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a depth estimation device, a depth estimation method, and a depth estimation program for estimating a depth value of a subject for a multi-view video from videos obtained by photographing the subject with a plurality of cameras.

  In recent years, research on generating and encoding stereoscopic video and free-viewpoint video using a plurality of videos (multi-view videos) obtained by shooting the same subject with a plurality of cameras is underway. In the technology for generating and encoding these stereoscopic video and free viewpoint video, the depth value to the subject is basically obtained by using the parallax amount of the camera video obtained by photographing the subject with a plurality of cameras ( (See Patent Document 1).

As shown in FIG. 13, this Patent Document 1 discloses a technique for encoding, in the time direction, camera images that are captured in a plurality of cameras (C ₁ , C ₂ , C ₃ ) and that have different parallax directions. Yes.
At the time of this encoding, the invention described in Japanese Patent Application Laid-Open No. H10-228867 is between camera images (F ₁ , F ₂ , F ₃ ) taken by a plurality of cameras (C ₁ , C ₂ , C ₃ ). Assuming the depth value of the subject in the video, the pixel value included in the block in the camera video (F ₂ ) captured by the reference camera (eg, C ₂ ), the positional relationship of the camera, and the depth value A depth value that minimizes the absolute value of the difference from the pixel value included in the corresponding block in the camera image (F ₁ , F ₃ ) captured by another fixed camera (eg, C ₁ , C ₃ ) is obtained. .

  Thus, conventionally, in a plurality of camera images, the pixel having the smallest difference absolute value of pixel values corresponding to a plurality of assumed depth values is a corresponding pixel of each camera image, and at that time, Was assumed to be the depth value of the pixel.

JP 2007-36800 A (paragraphs [0118] to [0112])

As described above, in the method of searching for a pixel having the smallest difference absolute value of pixel values corresponding to a plurality of assumed depth values in a plurality of camera images and estimating the depth value of the pixel, the following problems are encountered. There is.
For example, if there is no corresponding point between multiple camera images due to overlapping of the subjects, or multiple uniform responses with pixel values approximated between multiple camera images in a uniform subject with little change in pixel values There may be points. In these cases, the conventional method simply searches for a pixel with the smallest difference absolute value of the pixel value, and thus may find an incorrect corresponding point (false matching). That is, the conventional method has a problem that an incorrect depth value is estimated as a result of searching for an incorrect corresponding point.

  The present invention has been made in view of such a problem, and when there are no corresponding points between a plurality of camera images, or when there are a plurality of corresponding points between a plurality of camera images in a uniform subject. Even so, it is an object to provide a depth estimation device, a depth estimation method, and a depth estimation program that can reduce erroneous estimation of depth values.

  The present invention has been made to solve the above-mentioned problems. First, the depth estimation apparatus according to claim 1 is a plurality of cameras that photograph the same subject with a plurality of cameras arranged at predetermined positions. A depth estimation device for estimating a depth value indicating the depth of the subject from a video, comprising: a video input means; a corresponding pixel difference calculation means; a difference value determination means; a storage means; and a depth value determination means. It was set as the structure provided.

  In such a configuration, the depth estimation apparatus captures a subject with a reference image obtained by photographing a subject with a predetermined reference camera and a plurality of adjacent cameras adjacent to the reference camera in any of a plurality of cameras arranged by the image input unit. Input a plurality of adjacent videos. This adjacent video is a video for measuring the parallax from the reference video.

  Then, the depth estimation device uses a corresponding pixel difference calculation unit to calculate, for each of a plurality of assumed depth values, a plurality of adjacent images that are pixels corresponding to the parallax of the pixel value of each pixel of the reference image and the assumed depth value of the pixel. Next, an adjacent video pixel difference value that is a difference absolute value from the pixel value of each pixel is calculated. The assumed depth value is a value that assumes the depth value of the subject corresponding to the pixel of the reference image, and is, for example, a value from the minimum parallax (“0”) to the maximum parallax. By this corresponding pixel difference calculating means, an absolute difference value between pixel values of corresponding points corresponding to an assumed depth value between the pixel value of each pixel of the reference image and the pixel value of each pixel of the adjacent image is calculated. It can be said that the smaller the value of the difference absolute value (adjacent video pixel difference value), the higher the possibility that the assumed depth value points to the same corresponding point in the actual subject.

Further, the depth estimation device uses the difference value determination unit to determine whether the difference between the maximum difference value and the minimum difference value of the adjacent image pixel difference values is greater than a predetermined threshold value for each of the plurality of assumed depth values and the pixel positions of the reference image. Is smaller than the threshold value, the average difference value that is the average value of the adjacent video pixel difference values is selected. If it is larger than the threshold value, the minimum difference value of the adjacent video pixel difference values is selected and determined as the selected difference value. The average difference value may be an arithmetic average of adjacent video pixel difference values or a geometric average.
The depth estimation apparatus stores the selected difference value determined by the difference value determination unit in the storage unit in association with the assumed depth value and the pixel position of the reference image.

Note that if the selected difference value is the correct depth value, the value becomes small, and even if a certain adjacent video pixel difference value takes an incorrect value due to noise or the like, a plurality of adjacent video pixel difference values are obtained. By averaging, the selection difference value becomes smaller as a whole.
Further, even if a certain adjacent video pixel difference value becomes a large value due to occurrence of occlusion, the minimum difference value in the other adjacent video is selected by the threshold determination, so the selection difference value is It will be selected as a more correct value.
Thus, the selection difference value becomes an index for determining a correct depth value among the assumed depth values.
Then, the depth estimation apparatus searches the storage unit for an assumed depth value at which the selection difference value is minimum for each pixel position by the depth value determination unit, and determines the depth value at the pixel position.

The depth estimation apparatus according to claim 2 is a depth estimation apparatus that estimates a depth value indicating a depth of the subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position. Thus, the image input means, the corresponding pixel difference calculation means, the difference value determination means, the storage means, and the depth value determination means are provided.
The difference value determination means of the depth estimation apparatus according to claim 1 determines the selection difference value based on the difference between the maximum difference value and the minimum difference value of the adjacent video pixel difference values. The difference value determination means of the depth estimation apparatus determines the selection difference value based on the ratio between the maximum difference value and the minimum difference value of the adjacent video pixel difference values.

  That is, in such a configuration, the depth estimation device uses the difference value determination unit to determine in advance the ratio between the maximum difference value and the minimum difference value of the adjacent image pixel difference values for each of the plurality of assumed depth values and the pixel positions of the reference image. If it is within the defined threshold range, the average difference value that is the average value of the adjacent video pixel difference values is selected. If it is outside the range, the minimum difference value of the adjacent video pixel difference values is selected, and the selected difference value Determine as.

Note that if the selection difference value is the correct depth value, the value becomes small, and even if a certain adjacent video pixel difference value takes an incorrect value due to noise or the like, the ratio of the adjacent video pixel difference value is If within a predetermined range, the selection difference value becomes smaller as a whole by averaging them. Thus, the selection difference value becomes an index for determining a correct depth value among the assumed depth values.
In addition, even when the adjacent video pixel difference value becomes a large value and the ratio is large or small due to the occurrence of occlusion, the minimum difference value in the other adjacent video is selected by the threshold determination. Therefore, the selection difference value is selected as a more correct value.

  The depth estimation apparatus according to claim 3 is the depth estimation apparatus according to claim 1 or 2, wherein the difference value determining means includes a second threshold value that is set in advance in the adjacent video pixel difference value. When a smaller difference absolute value exists, the difference absolute value is determined as the selected difference value.

  In such a configuration, the depth estimation device does not perform the determination for selecting the average difference value or the minimum difference value by the difference value determining unit if the adjacent image pixel difference value is smaller than the second threshold value, and the adjacent image pixel difference value. Select itself. For example, when the value that can be taken by the pixel is set to a level between “0” and “255”, the second threshold is set to about several levels. This is because the difference value selected by the difference value determining means is likely to be a difference value corresponding to the correct depth value if it is about several levels.

  Furthermore, the depth estimation apparatus according to claim 4 is configured to further include a smoothing unit in the depth estimation apparatus according to any one of claims 1 to 3.

  In such a configuration, the depth estimation apparatus uses the smoothing unit to select a hypothetical depth value that minimizes the selection difference value and a selection difference value at an adjacent pixel position of the pixel position for each pixel position of the reference image. By weighting and adding the difference from the assumed depth value at which is the minimum, the selected difference value at the pixel position is smoothed. As a result, the depth value determining means searches for an assumed depth value at which the smoothed selection difference value is minimized, and the depth value itself is smoothed between adjacent pixels in order to obtain the depth value at the pixel position. Will be converted.

A depth estimation apparatus according to claim 5 is the depth estimation apparatus according to any one of claims 1 to 4, wherein an assumed depth value is smaller than a pixel interval of the camera image. The pixel interval is a unit, and the corresponding pixel difference calculation means interpolates from a pixel value of a pixel including the sub-pixel corresponding to the assumed depth value and a pixel value of a pixel adjacent to the pixel. The difference absolute value is calculated using the pixel values generated in this way.
In such a configuration, the depth estimation apparatus calculates the difference value in units of sub-pixels by the corresponding pixel difference calculation unit, and determines the depth value in units of sub-pixels.

  Furthermore, the depth estimation apparatus according to claim 6 is the depth estimation apparatus according to any one of claims 1 to 5, wherein the pixel value includes a plurality of luminance values, color difference values, or color values. The corresponding pixel difference calculation means calculates the difference absolute value as a sum of absolute differences of values for each element.

  In such a configuration, the depth estimation apparatus can calculate the difference between the luminance value and the color difference value even if the pixel value is a value composed of a plurality of elements such as a luminance value and a color difference value, or a color value (for example, RGB value). The difference absolute value can be processed as a single value as the sum of absolute values or the sum of absolute differences of RGB values.

  The depth estimation method according to claim 7 is a depth estimation method for estimating a depth value indicating the depth of the subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position. Thus, the procedure includes a video input step, a corresponding pixel difference calculation step, a difference value determination step, a storage step, and a depth value determination step.

In such a procedure, the depth estimation method is a video input step in which a reference video obtained by shooting a subject with a predetermined reference camera and a plurality of adjacent cameras adjacent to the reference camera are shot in any of a plurality of cameras arranged. The plurality of adjacent videos are input by video input means.
And the depth estimation method is a corresponding pixel difference calculation step, for each of a plurality of assumed depth values, a plurality of adjacent images that are pixels corresponding to the parallax of each pixel value of the reference image and the assumed depth value of the pixel. The adjacent pixel difference value, which is a difference absolute value from the pixel value of each pixel, is calculated by the corresponding pixel difference calculation means.

Further, in the depth estimation method, in the difference value determination step, the adjacent image pixel difference value calculated in the corresponding pixel difference calculation step is determined by the difference value determination unit for each of a plurality of assumed depth values and pixel positions of the reference image. When the difference between the maximum difference value and the minimum difference value of the adjacent video pixel difference values calculated in the difference calculation step is smaller than a predetermined threshold, an average difference value that is an average value of the adjacent video pixel difference values is selected. If it is greater than the threshold, the minimum difference value of the adjacent video pixel difference values is selected and determined as the selected difference value.
In the depth estimation method, in the storage step, the selection difference value determined in the difference value determination step is stored in the storage unit in association with the assumed depth value and the pixel position of the reference image.
Thereafter, in the depth estimation method, in the depth value determination step, the depth value determination unit searches the storage unit for an assumed depth value at which the selection difference value is minimum for each pixel position, and sets it as the depth value of the pixel position.

The depth estimation method according to claim 8 is a depth estimation method for estimating a depth value indicating a depth of the subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position. Thus, the procedure includes a video input step, a corresponding pixel difference calculation step, a difference value determination step, a storage step, and a depth value determination step.
In addition, although the difference value determination step of the depth estimation method according to claim 7 determines the selection difference value based on the difference between the maximum difference value and the minimum difference value of the adjacent video pixel difference values, In the difference value determination step of the depth estimation method, the selection difference value is determined based on the ratio between the maximum difference value and the minimum difference value of the adjacent video pixel difference values.

  That is, in such a procedure, the depth estimation method uses a difference value determination unit to determine in advance the ratio between the maximum difference value and the minimum difference value of the adjacent image pixel difference values for each of the plurality of assumed depth values and the pixel positions of the reference image. If it is within the defined threshold range, the average difference value that is the average value of the adjacent video pixel difference values is selected. If it is outside the range, the minimum difference value of the adjacent video pixel difference values is selected, and the selected difference value Determine as.

  The depth estimation program according to claim 9 is a computer for estimating a depth value indicating the depth of the subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position. Are configured to function as video input means, corresponding pixel difference calculation means, difference value determination means, and depth value determination means.

In such a configuration, the depth estimation program captures a subject with a reference image obtained by photographing a subject with a predetermined reference camera and a plurality of adjacent cameras adjacent to the reference camera in any of a plurality of cameras arranged by the image input means. Input a plurality of adjacent videos.
Then, the depth estimation program uses a corresponding pixel difference calculation unit to calculate, for each of a plurality of assumed depth values, a plurality of adjacent images that are pixels corresponding to the parallax of the pixel value of each pixel of the reference image and the assumed depth value of the pixel. Next, an adjacent video pixel difference value that is a difference absolute value from the pixel value of each pixel is calculated.

  Further, the depth estimation program causes the difference value determining means to determine that the difference between the maximum difference value and the minimum difference value of the adjacent video pixel difference values is greater than a predetermined threshold value for each of a plurality of assumed depth values and pixel positions of the reference image. Is also selected, the average difference value that is the average value of the adjacent video pixel difference values is selected, and if larger than the threshold value, the minimum difference value of the adjacent video pixel difference values is selected and determined as the selected difference value, Write to the storage means in association with the assumed depth value and the pixel position of the reference image. Then, the depth estimation program searches the storage unit for an assumed depth value at which the selection difference value is minimum for each pixel position by the depth value determination unit, and determines the depth value at the pixel position.

The depth estimation program according to claim 10 is a computer for estimating a depth value indicating the depth of a subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position. Are configured to function as video input means, corresponding pixel difference calculation means, difference value determination means, and depth value determination means.
The difference value determination means of the depth estimation program according to claim 9 determines the selection difference value based on the difference between the maximum difference value and the minimum difference value of the adjacent video pixel difference values. The difference value determination means of the depth estimation program determines the selection difference value based on the ratio between the maximum difference value and the minimum difference value of the adjacent video pixel difference values.

  In other words, in such a configuration, the depth estimation program causes the difference value determination unit to determine in advance the ratio between the maximum difference value and the minimum difference value of the adjacent video pixel difference values for each of the plurality of assumed depth values and the pixel positions of the reference image. If it is within the defined threshold range, the average difference value that is the average value of the adjacent video pixel difference values is selected. If it is outside the range, the minimum difference value of the adjacent video pixel difference values is selected, and the selected difference value Determine as.

The present invention has the following excellent effects.
According to the first, second, seventh and tenth aspects of the present invention, even when an occlusion where a part of the subject is hidden behind the subject in front and there is no corresponding point occurs, Difference value, that is, a more correct difference value is selected, so that it is possible to prevent an erroneous depth value from being estimated. In addition, according to the present invention, even if false matching that causes a corresponding point to be mistaken in a uniform subject occurs, the average value of the difference values is selected according to the threshold value, so that the correct difference value is It is not a small value, but the degree of selection as a correct difference value can be reduced, and an erroneous depth value can be prevented from being estimated.

According to the third aspect of the present invention, it is possible to directly select the adjacent video pixel difference value when the depth value is correct, so that it is possible to increase the estimation accuracy of the depth value.
According to the invention of claim 4, since the depth value of the pixel can be smoothed by the depth value of the adjacent pixel, even if the depth value is erroneously estimated due to noise or the like, Since smoothing is performed by the depth value, a smooth depth value can be obtained.

  According to the fifth aspect of the present invention, since the assumed step size of the depth value can be made smaller than the pixel interval of the video, even if the interval between the cameras is narrowed, the depth value The resolution can be increased. As a result, the camera interval can be reduced, the system configuration including the camera can be downsized, the occurrence of occlusion can be reduced, and the erroneous estimation of the depth value can be reduced. .

  According to the sixth aspect of the present invention, not only a single difference such as only a luminance value but also a difference absolute value sum of values for each of a plurality of elements is used as a difference absolute value, thereby improving the correspondence accuracy of pixels. It is possible to prevent the estimation of an erroneous depth value due to false matching.

It is a block diagram which shows the structure of the depth estimation system which concerns on embodiment of this invention. It is a block diagram which shows the structure of the depth estimation apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the processing content of the difference value determination means in positive matching. It is explanatory drawing for demonstrating the processing content of the difference value determination means in false matching. It is explanatory drawing for demonstrating the processing content of the difference value determination means at the time of occlusion generation | occurrence | production. It is explanatory drawing for demonstrating the memory content memorize | stored in a memory | storage means. It is a flowchart which shows the whole operation | movement of the depth estimation apparatus which concerns on embodiment of this invention. It is a flowchart which shows the difference value determination operation | movement of the depth estimation apparatus which concerns on embodiment of this invention. It is a flowchart which shows the difference value smoothing operation | movement of the depth estimation apparatus which concerns on embodiment of this invention. It is a flowchart which shows the depth value determination operation | movement of the depth estimation apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the depth estimation apparatus which concerns on the modification of embodiment of this invention. It is explanatory drawing for demonstrating the method of producing | generating the pixel value of a subpixel by interpolation in case an assumption depth value is a subpixel unit. It is explanatory drawing for demonstrating the conventional depth estimation method.

Embodiments of the present invention will be described below with reference to the drawings.
[Overall configuration of depth estimation system]
Initially, with reference to FIG. 1, the whole structure of the depth estimation system containing the depth estimation apparatus which concerns on embodiment of this invention is demonstrated. The depth estimation system S generates a depth video in which the depth value of a subject is estimated from a multi-view video obtained by photographing the subject with a plurality of cameras arranged. The depth estimation system S shown in FIG. 1 includes a camera C (C _L , C _C , C _R ) and a depth estimation device 1.

The camera C is a general photographing device that photographs the subject T. Here, it is possible to generate depth values in accordance with the horizontal parallax with respect to serving as a reference camera (reference camera) C _C parallax, and equally spaced by a distance L determined in advance in the horizontal direction, the camera (adjacent _cameras) C L, are arranged parallel to the _{C R.} The distance L is arbitrary as long as the depth value and the amount of parallax can be accommodated.

The camera C _C is for shooting the image (reference image F _C) as a reference of parallax. The cameras C _L and C _R are cameras arranged on the left and right sides of the camera C _C , respectively, and shoot images (adjacent images F _L and F _R ) for obtaining parallax with the reference image F _C. It is. Videos (F _L , F _C , F _R ) photographed by the camera C (C _L , C _C , C _R ) are input to the depth estimation device 1.

Note that if the depth value is generated according to the parallax in the vertical direction, the cameras C may be arranged apart from each other in the vertical direction.
In addition, the video (F _L , F _C , F _R ) captured by the camera C may be a still image or a moving image. In the case of a moving image, the camera C performs frame synchronization, and sequentially inputs the frame image of the still image to the depth estimation device 1. Alternatively, a time code may be added for each frame image, and the depth estimation apparatus 1 may be synchronized.

The depth estimation apparatus 1 estimates a depth value indicating the depth of the subject T from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged at predetermined positions. Here, the depth estimation apparatus 1 uses the camera video (F _L , F _C , F _R ) taken by the camera C (C _L , C _C , C _R ) arranged in the horizontal direction to determine the depth of the subject T. The depth value indicating is estimated. Here, the depth estimation apparatus 1, the depth value, and generating a depth image F _Z by associating to each pixel value of the image.

[Configuration of depth estimation device]
Next, referring to FIG. 2 (refer to FIG. 1 as appropriate), the configuration of the depth estimation apparatus according to the embodiment of the present invention will be described. As shown in FIG. 2, the depth estimation apparatus 1 includes a video input unit 10, a corresponding pixel difference calculation unit 20, a difference value determination unit 30, a storage unit 40, a smoothing unit 50, and a depth value determination unit 60. And.

The video input means 10 inputs a plurality of camera videos taken by the camera C. Here, the image input unit 10, predetermined the reference image _{F C} taken by the base camera _{C C} were, the base camera _C adjacent to the _C plurality of adjacent cameras _C L, _{C R} taken with neighboring image _F L, F _{And R.}
Each video (F _L , F _C , F _R ) input by the video input unit 10 is stored in a memory (not shown) and is referred to by a corresponding pixel difference calculation unit 20 described later.

For each of a plurality of assumed depth values, the corresponding pixel difference calculation means 20 calculates the pixel value of each pixel of the reference image F _C and the adjacent images F _L and F _R corresponding to the parallax of the assumed depth value of the pixel. The adjacent image pixel difference values e _CL and e _CR which are absolute differences from the pixel values are calculated. Here, the neighboring image pixel difference value e _CL is a calculated difference absolute value between the reference image F _C and the neighboring image F _L, the neighboring image pixel difference value e _CR is the reference image F _C and the neighboring image is a calculated difference absolute value between F _R.

Note that the absolute difference value of the pixel value is the luminance of the pixel when the camera image (F _L , F _C , F _R ) is composed of each element of luminance (luminance value) and color difference signal (color difference value). A value obtained by adding the difference absolute value of the value and the difference absolute value of the color difference value. At this time, the color component weight may be adjusted by multiplying the absolute difference value of the color difference value by an arbitrary coefficient.
In addition, when the camera image (F _L , F _C , F _R ) is composed of RGB color signals, the absolute difference value of the pixel value is the difference absolute value of the R value that is an element of the color signal, and the G value. And the difference absolute value of the B value are added to each other.

  The assumed depth value is a value that temporarily sets the depth value of the subject T, and takes a value from “0” to a depth value corresponding to a predetermined maximum parallax amount. For example, when the depth value is represented by 8-bit data, the virtual depth value takes a value in the range of “0” to “255”. Here, the corresponding pixel difference calculation means 20 sequentially sets the assumed depth value from “0” to the depth value corresponding to the maximum amount of parallax (for example, “255”), and sets the assumed depth value to the parallax. A difference in pixel value between corresponding pixels is calculated as an adjacent video pixel difference value. Here, the unit of the depth value is the same as the unit of the parallax between the pixels, and when the depth value increases by “1”, the parallax increases by one pixel.

Here, the corresponding pixel difference calculating means 20, the reference image _{F C} of (x, y) the pixel value of the coordinate _F C (x, y), the reference image _{F C} (x, y) coordinates of the hypothetical depth value d corresponding to (x + d, y) of the neighboring image _{F L} pixel values of the coordinates _F L (x + d, y), of the neighboring image _{F R (x-d, y} ) the pixel value of the coordinate _F R (x-d, When y), the adjacent video pixel difference values e _CL (d, x, y) and e _CR (d, x, y) are calculated by the calculation of the following equation (1). “||” indicates an absolute value.
e _CL (d, x, y) = | F _L (x + d, y) −F _C (x, y) |
e _CR (d, x, y) = | F _R (x−d, y) −F _C (x, y) | (1)

The corresponding pixel difference calculating unit 20 outputs the adjacent video pixel difference values e _CL and e _CR for each assumed depth value d (“0” to the maximum parallax amount) obtained by the calculation to the difference value determining unit 30. The corresponding pixel difference calculation means 20 is notified by the difference value determination means 30 that the difference value has been written in the storage means 40 (writing completion), whereby the pixel position and the assumed depth value of the pixel to be calculated are notified. Are sequentially updated, and adjacent video pixel difference values e _CL and e _CR are calculated.

Difference value determining means 30, for each pixel position of a plurality of hypothetical depth values and the reference image F _C, the calculated neighboring image pixel difference value e _CL at the corresponding pixel difference calculating unit _20, based on the e _CR, the assumption depth The adjacent video pixel difference value of the pixel position in the value is determined.

  Here, the difference value determination means 30 includes an average difference value calculation means 30a, a minimum difference value selection means 30b, and a difference value selection means 30c.

Average differential value calculating means 30a, for each pixel position of a plurality of hypothetical depth values and the reference image F _C, calculated by the corresponding pixel difference calculating unit 20 the neighboring image pixel difference value e _CL, the average value of e _CR (average difference The value ea) is calculated.
That is, the average difference value calculating unit 30a, the reference image _{F C} of (x, y) average difference value ea corresponds to a hypothetical depth value d of coordinates (d, x, y) the following (2) calculation of the equation Calculated by
ea (d, x, y) = {e _CL (d, x, y) + e _CR (d, x, y)} / 2
... (2) formula

  The average difference value calculating unit 30a outputs an average difference value ea for each assumed depth value d ("0" to the maximum parallax amount) obtained by the calculation to the difference value selecting unit 30c.

Minimum difference value selecting unit 30b, for each pixel position of a plurality of hypothetical depth values and the reference image F _C, corresponding pixel difference calculating unit 20 with the calculated neighboring image pixel difference value e _CL, among e _CR, the value The smaller one (minimum difference value em) is selected.
That is, the minimum difference value selection unit 30b, the reference image _{F C} of (x, y) the neighboring image pixel difference values corresponding to the assumption depth value d of the coordinate _{e CL (d, x, y} ) and, neighboring image pixel difference value e _CR (d, x, y) is compared, and the smaller value is selected as the minimum difference value em as shown in the following equation (3).
em (d, x, y) = min {e _CL (d, x, y), e _CR (d, x, y)}
... (3) formula

  The minimum difference value selection unit 30b outputs the minimum difference value em for each selected assumed depth value d (“0” to the maximum amount of parallax) to the difference value selection unit 30c.

Difference value selection unit 30c, for each pixel position of a plurality of hypothetical depth values and the reference image F _C, the calculated neighboring image pixel difference value e _CL at the corresponding pixel difference calculating unit _20, based on the magnitude of the difference e _CR Thus, either the average difference value ea calculated by the average difference value calculation unit 30a or the minimum difference value em selected by the minimum difference value selection unit 30b is used as a difference value ( This is selected as a candidate for (adjacent video pixel difference value).

That is, the difference value selection means 30c, as shown in the following equation (4), the difference between the adjacent video pixel difference values e _CL and e _CR calculated by the corresponding pixel difference calculation means 20, and a predetermined threshold th If the absolute value of the difference is smaller than the threshold th, the average difference value ea calculated by the average difference value calculation means 30a is selected. Otherwise, the minimum value selected by the minimum difference value selection means 30b is selected. The difference value em is selected.
| E _CL (d, x, y) −e _CR (d, x, y) | <th (4)

The threshold th can be arbitrarily determined, but is preferably set to a value of about 1/6 of the change range of the pixel value. When this threshold value th is too large, it determines the pixel of the reference image F _C, even if the false match between pixels of neighboring image F _L or neighboring image F _R is not generated, the false matching is occurring, the threshold th is too small, the pixel of the reference image F _C, even when occlusion does not occur between pixels of neighboring image F _L or neighboring image F _R, will be thus determined that the occlusion has occurred, since This is because an incorrect depth value is determined in the process. Therefore, for example, when the change range of the pixel value is “0” to “255”, the threshold th is preferably set to about “30” to “60”.

The difference value selection unit 30c determines that the difference between the adjacent image pixel difference values e _CL and e _CR calculated by the corresponding pixel difference calculation unit 20 is smaller than a predetermined threshold (second threshold). The value may be selected as a candidate for a difference value (adjacent video pixel difference value) of the pixel position in the assumed depth value. The second threshold value is smaller than the threshold th, and is, for example, “1” to “3”. This is because if one of the adjacent video pixel difference values e _CL and e _CR does not fall below the second threshold value, it is highly likely that the difference value itself indicates a difference value at the correct depth value.
Hereinafter, the difference value selected by the difference value selection unit 30c is particularly referred to as a selection difference value.

  The difference value selection unit 30c writes the selection difference value (adjacent video pixel difference value) in the storage unit 40 in association with the assumed depth value and the pixel position of the reference video. The difference value determination unit 30 notifies the corresponding pixel difference calculation unit 20 of the completion of writing every time the selected difference value is written in the storage unit 40. Further, the difference value determining means 30 notifies the smoothing means 50 of the completion of writing after the storage means 40 has written the selected difference value for each assumed depth value at all pixel positions of the reference video.

Here, with reference to FIG. 3 to FIG. 5, the processing content and the operation of the difference value determination unit 30 will be schematically described.
First, referring to FIG. 3 (appropriately with reference to Figure 2), in certain assumptions depth value, and the pixels of the reference image _{F C} (target pixel _{P C),} the neighboring image _F L, _{F R} of the pixel (corresponding pixel _P L, A process in a case where P _R ) is correctly matched (positive matching) will be described.

As shown in FIG. 3, when the pixels are correctly matched, the adjacent video pixel difference values (difference absolute values) e _CL and e _CR calculated by the corresponding pixel difference calculation means 20 are both less than several levels. A sufficiently small value. At this time, if the difference between the adjacent video pixel difference values (difference absolute values) e _CL and e _CR is smaller than the threshold th, the difference value selection unit 30c calculates the average difference value ea calculated by the average difference value calculation unit 30a. Are selected as the virtual depth value and the difference value (selection difference value) e at the pixel position.

In this case, since the difference value (selection difference value) e is the average of the difference absolute values e _CL and e _CR , it can be said that the variation of the camera is absorbed and the value is more correct. Further, since the difference absolute values e _CL and e _CR are correct matching, both are sufficiently small values less than several levels, and the difference value (selection difference value) e that is the average value is also a small value less than several levels. It becomes.
As described above, if the matching is correct in a certain assumed depth value, the difference value determining means 30 selects the average difference value ea so that the difference value remains small and further absorbs camera variations. A highly accurate difference value can be determined.

Next, referring to FIG. 4 (appropriately with reference to Figure 2), in certain assumptions depth value, and the pixels of the reference image F _C (target pixel P _C), the pixel of the neighboring image F _L or neighboring image F _R (corresponding pixel Processing in the case where false matching occurs between P _L and P _R ) will be described.

As shown in FIG. 4, when a uniform object having a target pixel P _C is (little change in pixel value), even assuming the depth value is not correct, either neighboring image, for example, of the neighboring image F _R since the neighboring image pixel difference values between the corresponding pixels P _R (absolute differences) e _CR becomes small, there are cases where false matching may occur.

At this time, not a perfect match, a difference absolute value e _CR is not the small value of about several levels difference (in the figure, "medium"), a difference absolute value e _CR and the difference absolute value e _CL Is often smaller than the threshold th. Therefore, the difference value selection unit 30c selects the average difference value ea calculated by the average difference value calculation unit 30a as the difference value (selection difference value) e at the virtual depth value and the pixel position.

In this case, the difference value (selected difference value) e is an average of the difference absolute values e _CL and e _CR , and thus becomes a value larger than the difference absolute value e _CR .
As described above, the difference value determination unit 30 selects the average difference value ea when a false matching occurs in a certain assumed depth value, so that the difference value is set to a value larger than the difference value in which the false matching occurs. And the degree of error of the difference value can be increased.

Next, referring to FIG. 5 (appropriately with reference to Figure 2), in certain assumptions depth value, and the pixels of the reference image F _C (target pixel P _C), the pixel of the neighboring image F _L or neighboring image F _R (corresponding pixel Processing when occlusion occurs in P _L , P _R ) will be described.

As shown in FIG. 5, when the position of the object corresponding to the pixel of interest P _C is (here, the neighboring image F _R) adjacent images one camera is taken in, hidden behind the foreground object, The adjacent video (here, the adjacent video _FL ) captured by the other camera is often not hidden, and when the assumed depth value is correct, the difference absolute value _eCL is sufficiently small, and the difference absolute value _eCR Is a large value.

In this case, the difference between the difference absolute value _{e CR} and the difference absolute value _{e CL} is larger than the threshold value th. Therefore, the difference value selection unit 30c selects the minimum difference value em selected by the minimum difference value selection unit 30b as the difference value (selection difference value) e at the virtual depth value and the pixel position. In this case, since the smaller one is selected instead of the average of the difference absolute value _eCR and the difference absolute value _eCL , the correct difference value e is selected.

  As described above, when the assumed depth value (assumed depth value) is correct, the difference value determining means 30 has a smaller difference value (selected difference value) e corresponding to the depth value, and the depth described later. In the value determining means 60, the degree to which the correct depth value is selected can be increased.

Moreover, neighboring image F _L, if the false match in either F _R occurs, the difference absolute value e _CL selected difference value is (selected difference value) _e, the average value of e _CR increases. Therefore, in this case, the difference value determining unit 30 can reduce the degree to which the assumed depth value is selected in the depth value determining unit 60 described later.

Moreover, neighboring image F _L, if the occlusion with either F _R is occurring, if correct virtual depth value, the minimum difference value em is sufficiently small selected difference value is (selected difference value) e. Therefore, in this case, the difference value determination unit 30 can increase the degree to which the assumed depth value is selected in the depth value determination unit 60 described later.
As described above, the difference value (selected difference value) determined (selected) by the difference value determining means 30 can be used as an index indicating that the assumed depth value is correct.
Returning to FIG. 2, the description of the configuration of the depth estimation device 1 will be continued.

  The storage unit 40 stores the difference value (selected difference value) determined (selected) by the difference value determination unit 30 in association with the assumed depth value and the pixel position of the reference image. For example, the storage unit 40 is a general storage medium such as a memory.

Here, the contents of the selection difference value stored in the storage means 40 will be described with reference to FIG. As illustrated in FIG. 6, the storage unit 40 stores a selection difference value for each (x, y) coordinate that is a pixel position of the reference image. This x coordinate value takes a value from “0” to the video width (number of horizontal pixels) X of the reference video, and the y coordinate value is a value from “0” to the video height (number of vertical pixels) Y of the reference video. Take.
In addition, the storage unit 40 further stores a selection difference value for each (x, y) coordinate that is a pixel position of the reference image for each assumed depth value d. This assumed depth value d takes a value from “0” to a value corresponding to the maximum parallax amount (maximum parallax amount D).
Returning to FIG. 2, the description of the configuration of the depth estimation device 1 will be continued.

  The smoothing means 50 smoothes the selection difference value at the pixel position for each pixel position of the reference image. The smoothing means 50 first sets an assumed depth value at which the selection difference value is minimum for each pixel position of the reference image as a depth value candidate. Next, for each pixel position of the reference image, the smoothing unit 50 weights and adds the difference between the depth value candidate at the adjacent pixel position and its own depth value candidate to the selection difference value at the depth value candidate at the pixel position. Thus, the selection difference value at the pixel position is smoothed. The smoothing unit 50 starts the operation when notified by the difference value determining unit 30 that the selected difference value for each assumed depth value has been written at all pixel positions of the reference image (writing completion). .

Here, the smoothing means 50 performs weighted addition on all depth value candidates stored in association with the pixel positions of the reference video and their selection difference values.
That is, the smoothing means 50, the reference image F _C in the storage means 40 is stored (x, y) selected difference value e corresponding to the coordinates of the depth value candidates d (d, x, y), the following The smoothed selection difference value ev (d, x, y) is calculated by weighting and adding the depth value difference between adjacent pixels according to equation (5).
ev (d, x, y) = e (d, x, y)
+ W {| d (x + 1, y) -d (x, y) | + | d (x, y + 1) -d (x, y) |}
... (5)

Incidentally, d (x, y) is the reference image _{F C} of interest (x, y) selecting the difference value of the coordinate is the minimum depth value candidate, d (x + 1, y ) is, (x + 1, y) coordinates Depth value candidates with a minimum selection difference value, d (x, y + 1) indicate depth value candidates with a minimum selection difference value of (x, y + 1) coordinates. Further, w is a weighting coefficient, and this coefficient w can be an arbitrary value. The coefficient w may be adjustable from the outside via an input unit (not shown).

  Thus, the depth value of each adjacent pixel is smoothed by weighting and adding the difference between the depth value candidates between adjacent pixels to the selected difference value. As a result, a smooth depth value can be obtained even when the depth values at which the selection difference value is minimum between adjacent pixels are different from each other due to noise or the like.

  The smoothing unit 50 writes the depth value that minimizes the smoothed selection difference value obtained by the calculation as a depth value candidate in the storage unit 40. Thereafter, the smoothing means 50 notifies the depth value determining means 60 of the completion of smoothing.

  The depth value determining unit 60 smoothes the selection difference value stored in the storage unit 40 in association with the assumed depth value and the pixel position of the reference image, using the value of the depth value candidate for each pixel position. The hypothetical depth value with the smallest selection difference value is searched for as a new depth value candidate. Here, the depth value determining means 60 searches for an assumed depth value at which the selection difference value smoothed by the smoothing means 50 is minimum, and sets it as the depth value at the pixel position.

Thus, the depth determining means 60, by re-determining a depth value candidates in all the pixel positions, to generate a depth image F _Z estimating the depth value of the subject.
The depth value determining unit 60 starts the operation when notified from the smoothing unit 50 that the smoothing of the selected difference value has been completed (smoothing completion).

  As described above, the depth estimation apparatus 1 can estimate the depth value of the subject using the difference value of the parallax in the adjacent video as an index.

[Operation of depth estimation device]
Next, with reference to FIGS. 7-10, operation | movement of the depth estimation apparatus which concerns on embodiment of this invention is demonstrated. Here, the overall operation of the depth estimation apparatus 1 will be described with reference to FIG. 7, and the detailed description will be described with reference to FIGS. 8 to 10.

(Overall operation)
First, the overall operation of the depth estimation apparatus 1 will be described with reference to FIG.
First, the depth estimation apparatus 1 uses the video input means 10 to capture a camera image of the same subject T with cameras (reference camera C _C , adjacent cameras C _L , C _R ) arranged in parallel on the straight line at equal intervals. (reference image _{F C,} the neighboring image _F L, _{F R)} to enter (step S1).

The depth estimation apparatus 1 by the corresponding pixel difference calculating unit 20 and difference value determination unit 30, for each hypothetical depth value, the pixel value of each pixel of the reference image F _C, corresponding to the disparity of the hypothetical depth value of the pixel The adjacent video pixel difference values e _CL and e _CR , which are absolute differences from the pixel values of the adjacent pixels F _L and F _R , are determined, and either the average value or the minimum value is determined (selected). (Step S2; differential value determination operation). The operation in step S2 will be described later with reference to FIG.

The depth estimation apparatus 1, the smoothing means 50, the reference image F _C (x, y) corresponding to the assumed depth value d of the coordinates to the selected selected difference value in step S2, the depth between adjacent pixels The selected difference value is smoothed by weighted addition of the value differences (step S3; difference value smoothing operation). The operation in step S3 will be described later with reference to FIG.

After that, the depth estimation apparatus 1 searches the depth value candidate having the smallest smoothed selection difference value for each pixel position (x, y) by the depth value determination unit 60, and sets the depth value at the pixel position. Determine (step S4; depth value determination operation). The operation of step S4 will be described later with reference to FIG.
Through the above operation, the depth estimation apparatus 1 estimates a depth value indicating the depth of the subject from a plurality of camera images.

(Differential value determination operation)
Next, referring to FIG. 8 (refer to FIGS. 1 and 2 as appropriate for the configuration), the difference value determining operation that is the operation of step S2 of FIG. 7 will be described in detail.

First, the depth estimation apparatus 1 initializes the assumed depth value d, the x-coordinate value of the camera image, and the y-coordinate value as variables (d = 0, x = 0, y = 0) (step S10).
Thereafter, the depth estimation apparatus 1 by the corresponding pixel difference calculating unit 20, the reference image F _C (x, y) and the pixel value of the pixel of the coordinates, the neighboring image F _L corresponding to the parallax assumptions depth value d of the pixel , difference absolute value in the form of the neighboring image pixel difference value _e CL between the pixel value of the pixel of _{F R,} computed by said _{e CR} (1) formula (step S11).

That is, the corresponding pixel difference calculating means 20, the pixel values F _{C (x,} y) in the reference image F _C to the adjacent image F pixel values of a position moved d pixel pixel positions to the right of _L F _L ( The adjacent video pixel difference value e _CL (d, x, y) which is an absolute difference value with respect to x + d, y) is calculated.
Further, the corresponding pixel difference calculating means 20, the pixel values F _{C (x,} y) in the reference image F _C to the adjacent image F pixel values of the position of the pixel position moves to the left d pixels _R F _R ( An adjacent video pixel difference value e _CR (d, x, y), which is an absolute difference value with respect to x−d, y), is calculated.

Then, the depth estimation apparatus 1 uses the average difference value calculation unit 30a of the difference value determination unit 30 to calculate the adjacent video pixel difference value e _CL and the adjacent video pixel difference value e _CR obtained in step S11 according to the equation (2). Is calculated to obtain an average difference value ea. Further, the depth estimation apparatus 1, the minimum difference value selection means 30b of the difference value determination unit 30, by the equation (3), minimizing the minimum value of the neighboring image pixel difference value e _CL and neighboring image pixel difference value e _CR The difference value em is selected (step S12). Note that either the calculation of the average difference value ea or the selection of the minimum difference value em may be performed first.
Thereafter, the depth estimation apparatus 1 calculates the difference between the adjacent video pixel difference value e _CL and the adjacent video pixel difference value e _CR obtained in step S11 by the difference value selection unit 30c of the difference value determination unit 30, and the difference Is smaller than a predetermined threshold th (step S13).

Then, when the difference between the neighboring image pixel difference value _{e CL} and neighboring image pixel difference value _{e CR} is smaller than the threshold value th (Yes in step S13), and the difference value selecting unit 30c, as the difference value, calculated in step S12 The average difference value ea is selected and written in the storage means 40 in association with the assumed depth value d and the pixel position (x, y) of the reference image (step S14).
On the other hand, when the difference between the adjacent video pixel difference value e _CL and the adjacent video pixel difference value e _CR is equal to or greater than the threshold th (No in step S13), the difference value selection unit 30c obtains the minimum value obtained in step S12 as the difference value. The difference value em is selected and written in the storage means 40 in association with the assumed depth value d and the pixel position (x, y) of the reference image (step S15).

Incidentally, after step S12, the difference value selecting unit 30c is neighboring image pixel difference value e _CL or any second threshold neighboring image pixel difference value e _CR (e.g., a few pixels) equal to or less than, the second Writing either the adjacent video pixel difference value e _CL or the adjacent video pixel difference value e _CR equal to or less than the threshold value in the storage means 40 in association with the assumed depth value d and the pixel position (x, y) of the reference video; (Not shown as a step). In this case, the difference value selection unit 30c skips the determination in step S13. As a result, the correct difference value in the assumed depth value d is written in the storage means 40.

Thereafter, the depth estimation apparatus 1 sequentially adds (increments) the assumed depth value d, which is a variable, the x-coordinate value of the camera image, and the y-coordinate value, until the assumed depth value d reaches the maximum amount of parallax (step S16, S17) The operation from step S11 to step S15 until the x-coordinate value becomes the width of the camera image (steps S18 and S19), and the y-coordinate value becomes the height of the camera image (steps S20 and S21). repeat.
By this difference value determination operation, the storage means 40 stores difference values (selected difference values) corresponding to a plurality of assumed depth values d (“0” to maximum parallax amount) for each pixel position (x, y) of the reference image. ) E is stored.

(Difference smoothing operation)
Next, the difference value smoothing operation, which is the operation of step S3 in FIG. 7, will be described in detail with reference to FIG.
First, the depth estimation apparatus 1 reinitializes (x = 0, y = 0) the x coordinate value and the y coordinate value of the camera image, which are variables (step S30).
The depth estimation apparatus 1, the smoothing means 50, (x, y) of the reference image F _C stored in the storage unit 40 and the coordinate in the coordinate adjacent to the coordinates, in a hypothetical depth value The assumed depth value with the smallest selection difference value is searched for as a depth value candidate d (step S31).

In other words, the smoothing unit 50 determines that the selection difference value of the assumed depth value d (x, y) and (x + 1, y) coordinates at which the selection difference value of the (x, y) coordinate of the reference image F _C is minimum is the minimum. Assumed depth value d (x + 1, y) and (x, y + 1) are searched for an assumed depth value d (x, y + 1) that minimizes the selected difference value, and set as the first depth value candidate d.

The depth estimation apparatus 1, the smoothing means 50, (x, y) of the reference image F _C in the storage unit 40 are stored selected difference value e (d corresponding to the coordinates of the depth value candidates d, x, The smoothed selection difference value ev (d, x, y) is calculated by weighting and adding the difference between the depth value candidates between adjacent pixels to y) according to the equation (5) (step S32).
Then, the depth estimation apparatus 1 writes the depth value candidate that minimizes the smoothed selection difference value ev calculated in step S32 into the storage unit 40 in association with the pixel position (x, y) of the reference image (step S33).

Thereafter, the depth estimation apparatus 1 sequentially adds (increments) the x-coordinate value and y-coordinate value of the camera image, which are variables, until the x-coordinate value becomes the width of the camera image (steps S34 and S35). Regarding the value, the operations from step S31 to step S33 are repeated until the height of the camera image is reached (steps S36 and S37).
By this difference value smoothing operation, the storage unit 40 stores a depth value candidate that minimizes the smoothed selection difference value ev for each pixel position (x, y) of the reference image.

(Depth value determination operation)
Next, with reference to FIG. 10 (refer to FIGS. 1 and 2 as appropriate for the configuration), the depth value determination operation that is the operation of step S4 of FIG. 7 will be described in detail.

First, the depth estimation apparatus 1 initializes the x-coordinate value and the y-coordinate value of the camera image, which are variables, respectively (x = 0, y = 0) (step S50).
Then, the depth estimation apparatus 1 uses the depth value determination means 60 to select the smoothed selection difference value for each pixel position (x, y) for the selection difference value e of the depth value candidate stored in the storage means 40. The assumed depth value d that minimizes ev is searched (step S51). The assumed depth value d searched in step S51 is determined (output) as a new depth value candidate of the subject at the pixel position (x, y) (step S52).

Thereafter, the depth estimation apparatus 1 sequentially adds (increments) the x-coordinate value and y-coordinate value of the camera image, which are variables, until the x-coordinate value reaches the width of the camera image (steps S53 and S54). Regarding the value, the operations of steps S51 and S52 are repeated until the height of the camera image is reached (steps S55 and S56). Thereby, the depth value candidates of the subject at all the pixel positions (x, y) are determined.
This depth value determination operation may be repeated any number of times using newly determined depth value candidates. As a result, a depth value candidate that is smoother can be obtained.

As mentioned above, although the structure and operation | movement of the depth estimation apparatus 1 which concern on embodiment of this invention were demonstrated, this invention is not limited to this embodiment.
Hereinafter, various modifications of the present invention will be described.

[Modification of Embodiment of Depth Estimation Device]
(Modification 1: Modification in which smoothing means is omitted)
First, as a first modification, the depth estimation apparatus 1 illustrated in FIG. 2 may be configured by omitting the smoothing unit 50.
Here, with reference to FIG. 11, the structure of the depth estimation apparatus which abbreviate | omitted the smoothing means is demonstrated. The depth estimation apparatus 1B shown in FIG. 11 omits the smoothing means 50 from the depth estimation apparatus 1 (see FIG. 2), and replaces the difference value determination means 30 and the depth value determination means 60 with the difference value determination means 30B and the depth value, respectively. The determination unit 60B is configured. About the same structure as the depth estimation apparatus 1, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The difference value determination unit 30B is similar to the difference value determination unit 30 described in FIG. 2 with respect to its internal configuration and internal operation, but the selected difference value for each assumed depth value is obtained at all pixel positions of the reference image. The difference is that the destination of notifying the completion of writing after writing to the storage means 40 is the depth value determining means 60B.

The depth value determining means 60B has the same basic function as the depth value determining means 60 described with reference to FIG. 2, except that the depth value determining means 60B operates when notified of completion of writing from the difference value determining means 30B. .
Further, the operation of the depth estimation apparatus 1B is an operation excluding step S3 among the operations of the depth estimation apparatus 1 described in FIG.

  By configuring and operating the depth estimation device 1B in this manner, the depth estimation device 1B has a noise that has a minimum selection difference value between adjacent pixels compared to the depth estimation device 1 described in FIG. Etc., the depth estimation may be wrong. However, the depth estimation apparatus 1B can obtain the same effect as the depth estimation apparatus 1 at high speed in an image with little noise degradation.

  That is, the depth estimation apparatus 1B estimates the depth value of a pixel having the smallest difference absolute value of pixel values corresponding to a plurality of assumed depth values even when there are a plurality of corresponding points in a uniform subject. Compared to the conventional method, the depth value when false matching occurs in a certain adjacent video by switching between the average difference absolute value or the minimum difference absolute value depending on the threshold. Can be removed from the estimation target, and when occlusion occurs, the depth value can be estimated from the correct difference value, and the depth value can be estimated with high accuracy.

(Modification 2: Modification in which pixels are sub-pixels)
In the present embodiment, the assumed depth value unit (step size) is the same as the unit of parallax between pixels (pixel interval), but the virtual value of the sub-pixel is smaller than the pixel interval. It is good also as a space | interval (for example, half pixel unit, 1/4 pixel unit). In this case, for example, the pixel value of the half pixel can be obtained by interpolation from the pixel value including the half pixel and the pixel value of the adjacent pixel.

For example, if the pixels are arranged as shown in FIG. _{12, (x, y) pv} 1 pixel value of the pixel _{P 1} of the coordinates, the (x, y-1) pixel value of the pixel _{P 2} coordinates When the pixel value of the pixel P _{3 at} the pv ₂ , (x−1, y) coordinates is pv ₃ and the pixel value of the pixel P _{4 at} the (x−1, y−1) coordinates is pv ₄ , the virtual (x -1 / 2, y-1/ 2) pixel values psv subpixels _{P S} of coordinates may be calculated by the following equation (6).
_{psv = (pv 1 + pv 2} + pv 3 + pv 4) / 4 ... (6) equation Here, the pixel _P 1 adjacent to the sub-pixels _{P S, P 2, P 3} , the pixel values by interpolation using _{P 4} While the obtained when the sub-pixel P _S is adjacent only to the P ₁ and P ₃ may perform interpolation using only pixel P ₁ and P _3.

In this case, it is assumed that the corresponding pixel difference calculation means 20 of the depth estimation device 1 (FIG. 2) calculates the adjacent video pixel difference values e _CL and e _CR in units of sub-pixels. In the difference value determining unit 30, the smoothing unit 50, the storage unit 40, and the depth value determining unit 60, the processing performed in units of pixels may be performed in units of sub-pixels.
Thus, by performing the unit of depth values at the virtual sub-pixel spacing, the reference camera C _C, to increase the resolution of the adjacent camera C _L, depth values even when the camera distance between C _R is narrow Can do.

(Modification 3: Modification of the number of cameras)
Moreover, although this embodiment demonstrated as what estimates the depth of a to-be-photographed object using three cameras, it is good also as estimating a depth using four or more cameras.
In this case, the camera image for which the depth value is to be obtained is set as the reference image, and the corresponding pixel difference calculation means 20 calculates the absolute value of the difference between the corresponding pixels between the reference image and the camera images taken by a plurality of cameras arranged on both sides of the camera. Calculate each. Then, the average difference value calculating means 30a of the difference value determining means 30 calculates the average value (average difference value) of the plurality of difference absolute values, and the minimum difference value selecting means 30b is the minimum value of the plurality of difference absolute values ( Select the minimum difference value. Then, the difference value selection means 30c selects either the average difference value or the minimum difference value. The difference value selection means 30c selects the average difference when the difference between the maximum value (maximum difference value) and the minimum value (minimum difference value) of the plurality of absolute difference values is smaller than a predetermined threshold value. A value is selected, otherwise, a minimum difference value may be selected.
Thus, the estimation accuracy of the depth value can be increased by increasing the number of cameras.

(Modification 4: Modification of camera arrangement interval)
Further, in this embodiment, the installation intervals of the cameras that shoot the subject are set to be equal intervals, but different camera intervals may be used.
In this case, the corresponding pixel difference calculating means 20, each of the neighboring image F _L, the F _R, for each camera distance from the base camera C _C, from the pixels of the reference image F _C, the positions of the pixels corresponding to the parallax, respectively determined, it may be operational and the pixel value of the pixel, a difference absolute value between the pixel value of the pixel of the reference image F _C. For example, in the present embodiment, the unit of the depth value is the same as the unit of the parallax between the pixels, and the parallax increases by one pixel by increasing the depth value by “1”. camera C _C shown in _1, the distance between C _R, camera C _C, When was the twice the distance between C _L, corresponding pixel difference calculating unit 20, instead of the equation (1), the following What is necessary is just to calculate a difference absolute value by (7) Formula.
e _CL (d, x, y) = | F _L (x + d, y) −F _C (x, y) |
e _CR (d, x, y) = | F _R (x−2d, y) −F _C (x, y) | (7) Thereby, the degree of freedom of the camera arrangement of the depth estimation system (FIG. 1). Can be increased.

(Modification 5: Modification of Corresponding Pixel Difference Calculation Unit)
Further, in the present embodiment, the corresponding pixel difference calculating means 20, the pixel value of each pixel of the reference image F _C, for each pixel of the neighboring image F _L, F _R corresponding to the parallax assumptions depth value of the pixel pixel The absolute value of the difference with the value is calculated, but the absolute value of the difference is not only the corresponding pixel but also the weighted average value of the absolute difference value of the pixel values in the block centered on the corresponding pixel. Also good. For example, the corresponding pixel difference calculation means 20 calculates the adjacent video pixel difference value e _CL (d, x, y) of the equation (1) by the following equation (8).
e _CL (d, x, y) = | F _L (x + d, y) −F _C (x, y) | × 1/2
+ | F _L (x + d + 1, y) −F _C (x + 1, y) | × 1/8
+ | F _L (x + d-1, y) -F _C (x-1, y) | × 1/8
+ | F _L (x + d, y + 1) −F _C (x, y + 1) | × 1/8
+ | F _L (x + d, y−1) −F _C (x, y−1) | × 1/8
... (8) Formula By this, the difference | error of the difference absolute value which generate | occur | produces by noise etc. can be reduced.

(Modification 6: Modification of Average Difference Value Calculation Unit)
In the present embodiment, the average difference value calculating unit 30a calculates the average value of the adjacent video pixel difference values e _CL and e _CR calculated by the corresponding pixel difference calculating unit 20 by arithmetic averaging. Even if it calculates | requires, there can exist an effect similar to this embodiment.
That is, the average difference value calculation means 30a may obtain the average difference value ea by the following equation (9) instead of the above equation (2).
ea (d, x, y) = √ {e _CL (d, x, y) × e _CR (d, x, y)}
... (9) Formula

  Note that, as shown in the modification 3, the average difference value calculating unit 30a, when inputting camera images with four or more cameras, causes the corresponding pixel difference calculating unit 20 to perform adjacent video pixel difference values (difference absolute values). 3 or more are calculated. In this case, when the number of the adjacent video pixel difference values is n, the average difference value calculating unit 30a may set the nth root of the product of the adjacent video pixel difference values as the average difference value ea.

(Modification 7: Modification of Difference Value Selection Unit)
Further, in the present embodiment, the difference value selection unit 30 c is either the average difference value ea or the minimum difference value em depending on the difference between the adjacent video pixel difference values e _CL and e _CR calculated by the corresponding pixel difference calculation unit 20. However, even if the selection is made according to the ratio of the adjacent video pixel difference values e _CL and e _CR , the same effect as that of the present embodiment can be obtained.

That is, the difference value selection unit 30c replaces the determination of the equation (4) with the ratio of the adjacent video pixel difference values e _CL and e _CR within a predetermined ratio range as shown in the following equation (10). In other cases, the average difference value ea may be selected. In other cases, it may be determined that the occlusion described with reference to FIG. 5 is highly likely to occur, and the minimum difference value em may be selected.

1 / th _R <e _CL (d, x, y) / e _CR (d, x, y) <th _R (10) This threshold th _R is a value greater than “1”, for example, “5” or so. When the threshold th _R is too large, the pixel of the reference image F _C, even if the false match between pixels of neighboring image F _L or neighboring image F _R does not occur, determines that the false match occurs, the threshold value when th _R is too small, the pixel of the reference image F _C, even when occlusion does not occur between pixels of neighboring image F _L or neighboring image F _R, would be attempting to determine the occlusion has occurred It is.

  In addition, as shown in the modification 3, the difference value selection unit 30c, when the camera image is input by four or more cameras, is calculated by the corresponding pixel difference calculation unit 20 to determine the adjacent video pixel difference value (difference absolute value). Three or more are calculated. In this case, the difference value selection unit 30c may use a ratio between the maximum value (maximum difference value) and the minimum value (minimum difference value) of the adjacent video pixel difference values.

(Modification 8: Modification of smoothing means)
In the present embodiment, the smoothing unit 50 selects the selection difference value e (d, x) corresponding to the assumed depth value d of the (x, y) coordinates of the reference image F _C as shown in the equation (5). , Y) is weighted and added to the absolute value of the depth value difference between two adjacent pixels. However, as shown in the following equation (11), the absolute value of the difference value between the four adjacent pixels is weighted and added. May be.
ev (d, x, y) = e (d, x, y)
+ W {| d (x + 1, y) -d (x, y) | + | d (x, y + 1) -d (x, y) |
+ | D (x-1, y) -d (x, y) | + | d (x, y-1) -d (x, y) |}
... (11) Formula

  The modification examples of the depth estimation device 1 and the depth estimation device 1B according to the embodiment of the present invention have been described above. However, the modification examples such as the depth estimation device 1 and the depth estimation device 1B have been described above with respect to general computers. It can be operated by a depth estimation program that functions as each means. The depth estimation program can also be distributed via a communication line, or can be recorded on a recording medium such as a CD-ROM for distribution.

  As described above, the depth estimation devices 1 and 1B have the smallest difference absolute value of pixel values corresponding to a plurality of assumed depth values even when there are a plurality of corresponding points in a uniform subject. Compared to the conventional method of estimating the depth value of a pixel, false matching occurs in a certain adjacent video by switching between selecting the average difference absolute value or selecting the minimum difference absolute value according to the threshold value. If it is, the depth value can be excluded from the estimation target. If occlusion occurs, the depth value can be estimated from the correct difference value, and the depth value can be estimated accurately.

DESCRIPTION OF SYMBOLS 1 Depth estimation apparatus 10 Image | video input means 20 Corresponding pixel difference calculation means 30 Difference value determination means 30a Average difference value calculation means 30b Minimum difference value selection means 30c Difference value selection means 40 Storage means 50 Smoothing means 60 Depth value determination means

Claims (10)

A depth estimation device that estimates a depth value indicating the depth of the subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position,
In any of the plurality of cameras arranged in the plurality, a reference image obtained by photographing the subject with a predetermined reference camera and a plurality of adjacent images obtained by photographing the subject with a plurality of adjacent cameras adjacent to the reference camera are input. Video input means;
For each of a plurality of assumed depth values, a difference absolute value between a pixel value of each pixel of the reference image and a pixel value of each pixel of the plurality of adjacent images that is a pixel corresponding to the parallax of the assumed depth value of the pixel. Corresponding pixel difference calculating means for calculating the adjacent video pixel difference value,
When the difference between the maximum difference value and the minimum difference value of the adjacent video pixel difference value is smaller than a predetermined threshold for each of the plurality of assumed depth values and the pixel position of the reference video, the adjacent video pixel difference A difference value determining means for selecting an average difference value that is an average value of the values, and selecting a minimum difference value of the adjacent video pixel difference values and determining the selected difference value when the difference is larger than the threshold;
Storage means for storing the selection difference value determined by the difference value determination means in association with the assumed depth value and the pixel position of the reference image;
A depth value determining unit that searches the storage unit for the assumed depth value at which the selection difference value is minimum for each pixel position, and sets the depth value of the pixel position;
A depth estimation apparatus comprising: A depth estimation device that estimates a depth value indicating the depth of the subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position,
In any of the plurality of cameras arranged in the plurality, a reference image obtained by photographing the subject with a predetermined reference camera and a plurality of adjacent images obtained by photographing the subject with a plurality of adjacent cameras adjacent to the reference camera are input. Video input means;
For each of a plurality of assumed depth values, a difference absolute value between a pixel value of each pixel of the reference image and a pixel value of each pixel of the plurality of adjacent images that is a pixel corresponding to the parallax of the assumed depth value of the pixel. Corresponding pixel difference calculating means for calculating the adjacent video pixel difference value,
If the ratio between the maximum difference value and the minimum difference value of the adjacent video pixel difference values is within a predetermined threshold range for each of the plurality of assumed depth values and the pixel position of the reference video, the adjacent video pixels Selecting an average difference value that is an average value of the difference values, and, if out of the range, selecting a minimum difference value of the adjacent video pixel difference value, and determining a difference value determining means for determining the selected difference value;
Storage means for storing the selection difference value determined by the difference value determination means in association with the assumed depth value and the pixel position of the reference image;
A depth value determining unit that searches the storage unit for the assumed depth value at which the selection difference value is minimum for each pixel position, and sets the depth value of the pixel position;
A depth estimation apparatus comprising:   The difference value determining means determines the difference absolute value as the selection difference value when a difference absolute value smaller than a predetermined second threshold exists in the adjacent video pixel difference value. The depth estimation apparatus according to claim 1 or 2. For each pixel position, the difference between the assumed depth value at which the selection difference value is minimum and the assumed depth value at which the selection difference value at the adjacent pixel position of the pixel position is minimum is determined as the selection difference value. Smoothing means for smoothing the selection difference value at the pixel position by weighted addition;
The depth value determination unit searches for the assumed depth value that minimizes the selection difference value smoothed by the smoothing unit, and sets the depth value of the pixel position as the depth value. Depth estimation apparatus as described in any one of these. The assumed depth value is a value in units of virtual sub-pixel intervals smaller than the pixel interval of the camera image,
The corresponding pixel difference calculation means uses the pixel value generated by interpolation from the pixel value of the pixel including the sub-pixel corresponding to the assumed depth value and the pixel value of the pixel adjacent to the pixel, and the absolute difference value The depth estimation apparatus according to any one of claims 1 to 4, wherein the depth estimation device is calculated. The pixel value is a value composed of a plurality of elements of luminance value and color difference value, or color value,
The depth estimation device according to any one of claims 1 to 5, wherein the corresponding pixel difference calculation unit calculates the difference absolute value as a sum of absolute differences of values for each element. . A depth estimation method for estimating a depth value indicating the depth of a subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position,
Video input of a reference image obtained by photographing the subject with a predetermined reference camera and a plurality of adjacent images obtained by photographing the subject with a plurality of adjacent cameras adjacent to the reference camera in any of the plurality of cameras arranged in the plurality A video input step for inputting by means;
For each of a plurality of assumed depth values, a difference absolute value between a pixel value of each pixel of the reference image and a pixel value of each pixel of the plurality of adjacent images that is a pixel corresponding to the parallax of the assumed depth value of the pixel. A corresponding pixel difference calculation step for calculating the adjacent video pixel difference value by the corresponding pixel difference calculation means;
When a difference between the maximum difference value and the minimum difference value of the adjacent image pixel difference values is smaller than a predetermined threshold for each of the plurality of assumed depth values and the pixel position of the reference image by the difference value determination unit. A difference value that selects an average difference value that is an average value of the adjacent video pixel difference values and, when larger than the threshold, selects a minimum difference value of the adjacent video pixel difference values and determines it as a selection difference value A decision step;
A storage step of storing the selected difference value determined in the difference value determination step in a storage unit in association with the assumed depth value and the pixel position of the reference image;
A depth value determining step by searching the storage means for the assumed depth value at which the selection difference value is minimum for each pixel position by the depth value determining means, and setting the depth value at the pixel position;
The depth estimation method characterized by including. A depth estimation method for estimating a depth value indicating the depth of a subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position,
Video input of a reference image obtained by photographing the subject with a predetermined reference camera and a plurality of adjacent images obtained by photographing the subject with a plurality of adjacent cameras adjacent to the reference camera in any of the plurality of cameras arranged in the plurality A video input step for inputting by means;
For each of a plurality of assumed depth values, a difference absolute value between a pixel value of each pixel of the reference image and a pixel value of each pixel of the plurality of adjacent images that is a pixel corresponding to the parallax of the assumed depth value of the pixel. A corresponding pixel difference calculation step for calculating the adjacent video pixel difference value by the corresponding pixel difference calculation means;
When the difference value determining means determines that the ratio between the maximum difference value and the minimum difference value of the adjacent image pixel difference values is within a predetermined threshold range for each of the plurality of assumed depth values and the pixel position of the reference image. Selects an average difference value, which is an average value of the adjacent video pixel difference values, and, if out of the range, selects a minimum difference value of the adjacent video pixel difference values and determines the selected difference value as a difference value A decision step;
A storage step of storing the selected difference value determined in the difference value determination step in a storage unit in association with the assumed depth value and the pixel position of the reference image;
A depth value determining step by searching the storage means for the assumed depth value at which the selection difference value is minimum for each pixel position by the depth value determining means, and setting the depth value at the pixel position;
The depth estimation method characterized by including. In order to estimate a depth value indicating the depth of the subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged at predetermined positions, a computer is provided.
In any of the plurality of cameras arranged in the plurality, a reference image obtained by photographing the subject with a predetermined reference camera and a plurality of adjacent images obtained by photographing the subject with a plurality of adjacent cameras adjacent to the reference camera are input. Video input means,
For each of a plurality of assumed depth values, a difference absolute value between a pixel value of each pixel of the reference image and a pixel value of each pixel of the plurality of adjacent images that is a pixel corresponding to the parallax of the assumed depth value of the pixel. Corresponding pixel difference calculation means for calculating the adjacent video pixel difference value,
When the difference between the maximum difference value and the minimum difference value of the adjacent video pixel difference value is smaller than a predetermined threshold for each of the plurality of assumed depth values and the pixel position of the reference video, the adjacent video pixel difference An average difference value that is an average value of values is selected, and if it is greater than the threshold value, a minimum difference value of the adjacent video pixel difference value is selected and determined as a selection difference value, and the assumed depth value and the reference Difference value determining means for writing to the storage means in association with the pixel position of the video,
A depth value determining unit that searches the storage unit for the assumed depth value that minimizes the selection difference value for each pixel position, and sets the depth value of the pixel position as a depth value.
Depth estimation program characterized by functioning as In order to estimate a depth value indicating the depth of the subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged at predetermined positions, a computer is provided.
In any of the plurality of cameras arranged in the plurality, a reference image obtained by photographing the subject with a predetermined reference camera and a plurality of adjacent images obtained by photographing the subject with a plurality of adjacent cameras adjacent to the reference camera are input. Video input means,
For each of a plurality of assumed depth values, a difference absolute value between a pixel value of each pixel of the reference image and a pixel value of each pixel of the plurality of adjacent images that is a pixel corresponding to the parallax of the assumed depth value of the pixel. Corresponding pixel difference calculation means for calculating the adjacent video pixel difference value,
If the ratio between the maximum difference value and the minimum difference value of the adjacent video pixel difference values is within a predetermined threshold range for each of the plurality of assumed depth values and the pixel position of the reference video, the adjacent video pixels An average difference value that is an average value of difference values is selected, and if it is out of the range, a minimum difference value of the adjacent video pixel difference value is selected and determined as a selected difference value, and the assumed depth value and the reference Difference value determining means for writing to the storage means in association with the pixel position of the video,
A depth value determining unit that searches the storage unit for the assumed depth value that minimizes the selection difference value for each pixel position, and sets the depth value of the pixel position as a depth value.
Depth estimation program characterized by functioning as

Images