JP5565839B2 - Depth estimation apparatus, depth estimation method, and depth estimation program - Google Patents

Description

  The present invention relates to a depth estimation device, a depth estimation method, and a depth estimation program for estimating a depth value of a subject for a multi-view video from videos obtained by photographing the subject with a plurality of cameras.

  In recent years, research on generating and encoding stereoscopic video and free-viewpoint video using a plurality of videos (multi-view videos) obtained by shooting the same subject with a plurality of cameras is underway. In the technology for generating and encoding these stereoscopic video and free viewpoint video, the depth value to the subject is basically obtained by using the parallax amount of the camera video obtained by photographing the subject with a plurality of cameras ( (See Patent Document 1).

As shown in FIG. 20, this Patent Document 1 discloses a technique for encoding, in the time direction, camera images that are captured in a plurality of cameras (C ₁ , C ₂ , C ₃ ) and that have different parallax directions. Yes.
At the time of this encoding, the invention described in Japanese Patent Application Laid-Open No. H10-228867 is between camera images (F ₁ , F ₂ , F ₃ ) taken by a plurality of cameras (C ₁ , C ₂ , C ₃ ). Assuming the depth value of the subject in the video, the pixel value included in the block in the camera video (F ₂ ) captured by the reference camera (eg, C ₂ ), the positional relationship of the camera, and the depth value A depth value that minimizes the absolute value of the difference from the pixel value included in the corresponding block in the camera image (F ₁ , F ₃ ) captured by another fixed camera (eg, C ₁ , C ₃ ) is obtained. .

  As described above, in the conventional technique, in a plurality of camera images, a pixel having a minimum difference absolute value of pixel values corresponding to a plurality of assumed depth values is a corresponding pixel of each camera image, and The assumed depth value at that time was estimated to be the depth value of the pixel.

JP 2007-36800 A (paragraphs [0118] to [0112])

As described above, in the method of searching for a pixel having the smallest difference absolute value of pixel values corresponding to a plurality of assumed depth values in a plurality of camera images and estimating the depth value of the pixel, the following problems are encountered. There is.
For example, if there is no corresponding point between multiple camera images due to overlapping of the subjects, or multiple uniform responses with pixel values approximated between multiple camera images in a uniform subject with little change in pixel values There may be points. In these cases, the conventional method simply searches for a pixel with the smallest difference absolute value of the pixel value, and thus may find an incorrect corresponding point (false matching). Furthermore, the conventional method may search for false corresponding points (false matching) when the texture of the subject is uniform, and erroneously determine the presence or absence of occlusion when the subject is partly different in color. There are things to do.

The present invention has been made in view of such a problem. In the case where there are no corresponding points between a plurality of camera images, there is a case where there are a plurality of corresponding points between a plurality of camera images in a uniform subject. It is an object of the present invention to provide a depth estimation device, a depth estimation method, and a depth estimation program that can reduce erroneous estimation of depth values.
Furthermore, the present invention provides a depth estimation device, a depth estimation method, and a depth estimation device that can reduce erroneous estimation of a depth value even when the texture of the subject is uniform or when the subject has different colors. It is an object to provide a depth estimation program.

  The present invention has been made to solve the above-mentioned problems. First, the depth estimation device according to the first invention of the present application is a plurality of cameras that photograph the same subject with a plurality of cameras arranged in a predetermined position. A depth estimation device that estimates a depth value indicating the depth of a subject from an image, the image input means, a corresponding pixel difference calculation means, a neighboring pixel value output means, a neighboring depth corresponding pixel value output means, and a difference A value determining unit, a storage unit, a smoothing unit, a depth value determining unit, and a neighborhood depth value output unit are provided.

  In such a configuration, the depth estimation apparatus uses a video input unit to capture a reference image obtained by capturing the subject with a predetermined reference camera in any of the plurality of cameras arranged, and the subject with an adjacent camera other than the reference camera. Input a plurality of adjacent images taken of. This adjacent video is a video for measuring the parallax from the reference video, and is, for example, a video taken by two adjacent cameras positioned on the left and right with respect to the reference camera.

  Then, the depth estimation device uses the corresponding pixel difference calculation unit to calculate the pixel value of the pixel of interest and the assumed depth value of the pixel of interest for each pixel of interest of the assumed depth value assuming the depth value and the reference image. A corresponding pixel difference value that is an absolute difference value with respect to the pixel values of the corresponding pixels of the plurality of adjacent videos at the pixel position corresponding to the parallax is calculated. The assumed depth value is a value that assumes the depth value of the subject corresponding to the pixel of the reference image, and is, for example, a value from the minimum parallax (“0”) to the maximum parallax. By this corresponding pixel difference calculation means, an absolute difference value between the pixel value of the target pixel of the reference image and the corresponding pixel value corresponding to the assumed depth value of the adjacent image is calculated. It can be said that the smaller the value of the difference absolute value (corresponding pixel difference value of the adjacent video), the higher the possibility that the assumed depth value points to the same corresponding point in the actual subject.

  Then, the depth estimation device includes a pixel value of the target pixel of the reference video and a predetermined number of neighboring pixels left and right from the target pixel in the reference image input to the video input unit by the neighboring pixel value output unit. A pixel value of a neighboring pixel at a distant pixel position is output. Furthermore, the depth estimation apparatus corresponds to the parallax of the depth value of the neighboring pixels from the neighboring video on the left and right with respect to the reference video for each of the assumed depth value and the target pixel by the neighboring depth corresponding pixel value output unit. The pixel value of the neighborhood depth corresponding pixel at the pixel position is output.

  Then, the depth estimation device uses the difference value determination unit to calculate an average difference value that is an average value of the corresponding pixel difference values and a minimum value that is a minimum value of the corresponding pixel difference values for each of the assumed depth value and the target pixel. A difference value is obtained, and based on the pixel value of the target pixel, the pixel value of the neighboring pixel, the depth value of the neighboring pixel, and the pixel value of the neighboring depth corresponding pixel, either the minimum difference value or the average difference value Is determined as a selection difference value according to a predetermined condition. Furthermore, the depth estimation apparatus stores the selection difference value determined by the difference value determination unit in association with the assumed depth value and the pixel position of the reference image by the storage unit.

Note that if the selected difference value is the correct depth value, the value becomes small, and even if the corresponding pixel difference value of a certain adjacent video takes an incorrect value due to noise or the like, a plurality of corresponding pixel difference values are obtained. As a result, the selection difference value becomes smaller as a whole.
In addition, the depth estimation device has occlusion, so that even when a corresponding pixel difference value becomes a large value or when there are a plurality of corresponding points for a uniform subject, the depth estimation device The optimum value is selected from the average difference value and the minimum difference value. That is, the depth estimation apparatus selects the minimum difference value, that is, the difference value on the side without occlusion as the selection difference value.

  Here, if the texture of the subject is uniform, or if the subject is a part of a different color, the depth value can be calculated simply by using the pixel values and depth values of neighboring pixels acquired from the same reference image as the target pixel. The estimation accuracy is lowered. For this reason, the depth estimation apparatus uses the neighborhood depth corresponding pixel acquired from the adjacent image.

Then, the depth estimation apparatus uses the smoothing unit to calculate, for each target pixel, a hypothetical depth value that minimizes the selection difference value of the pixel adjacent to the target pixel and the target differential value of the target pixel. A smoothing evaluation value that is a sum of the smoothed difference values is calculated by calculating a smoothed difference value obtained by smoothing the selected difference value by weighting and adding a difference with an assumed depth value at which the selected difference value is minimized. Is calculated.
As a result, the smoothing means performs smoothing on all pixels of the reference video, so that the depth value changes smoothly in the same subject, and the erroneous estimation of the depth value is reduced.

Then, the depth estimation apparatus replaces the assumed depth value corresponding to the selection difference value stored in the storage means with a second assumed depth value in which a depth value different from the assumed depth value is assumed by the depth value determination means. When the recalculated evaluation value corresponding to the smoothed evaluation value is calculated and the recalculated evaluation value is less than the smoothed evaluation value calculated by the smoothing means, the second hypothetical depth value is calculated from the reference image. If the recalculated evaluation value is not less than the smoothed evaluation value, the assumed depth value corresponding to the selection difference value stored in the storage means is determined as the depth value of each pixel of the reference image. Determine as. This second hypothetical depth value is a value that assumes the depth value of the subject corresponding to the pixel of the reference image, as with the assumed depth value, and is, for example, a value from the minimum parallax amount (“0”) to the maximum parallax amount. And
That is, the depth value determining means calculates again whether or not there is a second assumed depth value at which the smoothed evaluation value is minimum in addition to the already calculated assumed depth value, and when this second assumed depth value exists. The second assumed depth value is determined as the assumed depth value. This makes the assumed depth value more accurate.

  Further, in the depth estimation device according to the second invention of the present application, the difference value determining means is configured such that the difference absolute value of the depth value between the neighboring pixels is equal to or less than a predetermined first threshold value, or between the neighboring pixels. When the difference absolute value of the pixel values is equal to or smaller than a predetermined second threshold, the average difference value is determined as the selection difference value.

  In such a configuration, when the difference between the pixel value and the depth value of the neighboring pixels is small by the difference value determination unit, the depth estimation device determines that there is no boundary between the foreground and the background in the vicinity of the target pixel, and calculates the average difference value. Select as the selected difference value. Thereby, the depth estimation apparatus can select a more correct selection difference value.

  Further, in the depth estimation apparatus according to the third invention of the present application, the difference value determining means has a difference absolute value of a depth value between the neighboring pixels exceeding a predetermined first threshold, and the pixels between the neighboring pixels. If the difference absolute value of the values exceeds a predetermined second threshold, the difference absolute value between the pixel value of the neighboring depth corresponding pixel and the pixel value of the target pixel is calculated from each of the left and right adjacent images with respect to the reference image. And calculating the absolute value of the difference between the pixel value of the neighboring depth corresponding pixel on the side having the smaller depth value of the left and right neighboring pixels and the pixel value of the target pixel, on the opposite side to the smaller one The minimum difference value is determined as the selected difference value when the difference is smaller than the absolute value of the difference between the pixel value of the neighboring depth corresponding pixel and the pixel value of the target pixel.

  In such a configuration, the depth estimation device, when the difference value determining unit causes the pixel value of the target pixel to be closer to the pixel value of the neighboring depth corresponding pixel on the side where the background is present than the pixel value of the neighboring depth corresponding pixel on the opposite side, It is determined that there is occlusion, and the minimum difference value is selected as the selected difference value. Thereby, the depth estimation apparatus can select a more correct selection difference value.

  Further, in the depth estimation apparatus according to the fourth invention of the present application, the difference value determining means has a difference absolute value of a depth value between the neighboring pixels exceeding a predetermined first threshold value, and a pixel between the neighboring pixels. If the difference absolute value of the values exceeds a predetermined second threshold, the difference absolute value between the pixel value of the neighboring depth corresponding pixel and the pixel value of the target pixel is calculated from each of the left and right adjacent images with respect to the reference image. And calculating the absolute value of the difference between the pixel value of the neighboring depth corresponding pixel on the side having the smaller depth value of the left and right neighboring pixels and the pixel value of the target pixel, on the opposite side to the smaller one The average difference value is determined as the selection difference value when it is not smaller than the absolute difference value between the pixel value of the neighboring depth corresponding pixel and the pixel value of the target pixel.

  In such a configuration, the depth estimation device, when the difference value determining unit causes the pixel value of the target pixel to be closer to the pixel value of the neighboring depth corresponding pixel on the opposite side than the pixel value of the neighboring depth corresponding pixel on the side where the background exists, It is determined that there is no occlusion, and the average difference value is selected as the selected difference value. Thereby, the depth estimation apparatus can select a more correct selection difference value.

  Further, in the depth estimation device according to the fifth invention of the present application, the difference value determination means determines the average difference value as the selected difference value when the depth value is estimated for the first time, and estimates the depth value. When the number of times is the second or later, the selection difference value is determined according to the predetermined condition.

  In such a configuration, the depth estimation apparatus determines the average difference value as the selection difference value in the first depth estimation, thereby preventing false matching and obtaining a correct depth value even for a subject with a uniform texture. presume. And the depth estimation apparatus can determine the presence or absence of occlusion correctly by performing the depth estimation after the 2nd time using the obtained depth value.

  Further, in the depth estimation device according to the sixth invention of the present application, when the corresponding pixel difference calculation means deviates from the adjacent video located on the left and right of the reference video, the corresponding pixel is determined with respect to the reference video. The difference absolute value of the pixel value of the corresponding pixel is calculated as the corresponding pixel difference value between the adjacent image on the opposite side to the adjacent image on the detachment side and the adjacent video further adjacent to the adjacent image on the opposite side. And

  In such a configuration, the depth estimation apparatus calculates the corresponding pixel difference value by using the corresponding pixel difference calculation means, using two adjacent images in which the corresponding pixels corresponding to the target pixel in the reference image are in the image. Thereby, the depth estimation device can prevent false matching.

  Further, in the depth estimation device according to the seventh invention of the present application, when the neighboring depth value output means has a difference absolute value of a depth value between the neighboring pixels equal to or smaller than a predetermined threshold (first threshold), the neighboring pixels The number is halved, and the depth value of a neighboring pixel at a pixel position that is separated from the pixel of interest by the number of neighboring pixels is output.

  In such a configuration, the depth estimation device has a large amount of parallax, so even if it is a subject far away in the reference video, it determines whether or not there is occlusion and even if it is determined that there is no occlusion. Then, the distance to the neighboring pixels is halved, and the presence / absence of occlusion in the adjacent video is determined again. Accordingly, the depth estimation apparatus can correctly determine the presence or absence of occlusion in the adjacent video even for a subject (eg, a pillar) with a narrow left and right width.

  The depth estimation method according to the eighth invention of the present application is a depth estimation method for estimating a depth value indicating the depth of a subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position. A video input step, a corresponding pixel difference calculation step, a difference value determination step, a smoothing step, and a depth value determination step, wherein the difference value determination step includes a neighboring pixel value output step, a neighboring depth value The procedure includes an output step and a neighborhood depth corresponding pixel value output step.

  In this procedure, the depth estimation method includes a reference image obtained by photographing the subject with a predetermined reference camera in any one of the plurality of cameras arranged in the image input step, and an image other than the reference camera. A plurality of adjacent images obtained by photographing the subject with an adjacent camera are input. Then, in the corresponding pixel difference calculating step, the corresponding pixel difference calculating means is configured such that the corresponding pixel difference calculating means determines the pixel value of the target pixel and the target pixel for each of the assumed depth value assuming the depth value and the target pixel of the reference image. A corresponding pixel difference value, which is an absolute difference value with respect to the pixel values of the corresponding pixels of the plurality of adjacent images at the pixel position corresponding to the parallax of the assumed depth value of the pixel, is calculated.

Then, in the depth estimation method, in the difference value determination step, the difference value determination means includes, for each of the assumed depth value and the target pixel, an average difference value that is an average value of the corresponding pixel difference values, and the corresponding pixel difference value. determining the minimum difference value is a minimum value of the pixel value of the pixel of interest, and the pixel values of the neighboring pixels from only the target pixel number neighboring pixels to the pixel position apart horizontally a predetermined depth of the neighboring pixel The minimum difference value or the average difference value is selected according to a predetermined condition based on the value and the pixel value of the neighboring depth corresponding pixel at the pixel position corresponding to the parallax of the depth value of the neighboring pixel. The difference value is determined and the selected difference value is written in the storage means in association with the assumed depth value and the pixel position of the reference image.

  Then, in the depth estimation method, in the smoothing step, for each pixel of interest, the smoothing unit assumes an assumed depth value at which the selection difference value of the pixel adjacent to the pixel of interest is the smallest in the selection difference value of the pixel of interest. And calculating the smoothed difference value obtained by smoothing the selected difference value by weighting and adding the difference between the selected depth value and the assumed depth value at which the selected difference value of the target pixel is minimum. A certain smoothing evaluation value is calculated.

  In the depth estimation method, in the depth value determination step, the depth value determination means assumes that the assumed depth value corresponding to the selection difference value stored in the storage means is a depth value different from the assumed depth value. When the recalculated evaluation value corresponding to the smoothed evaluation value is calculated by replacing with the assumed depth value, and the recalculated evaluation value is less than the smoothed evaluation value calculated by the smoothing means, the second assumed depth A value is determined as the depth value of each pixel of the reference image, and if the recalculated evaluation value is not less than the smoothing evaluation value, an assumed depth value corresponding to the selection difference value stored in the storage means is determined as the reference image Is determined as the depth value of each pixel.

In the depth estimation method, in the neighborhood depth value output step, the neighborhood depth value output means outputs the depth value of the neighborhood pixel. Further, the depth estimation method, the neighboring pixel value output step, the neighboring pixel value output means, for outputting the pixel value of the target pixel of the reference image in the reference image and the pixel values of the neighboring pixels.

The depth estimation method, in the vicinity depth corresponding pixel value output step, the proximity depth corresponding pixel value output means, for each of the pixel of interest the assumption depth value and, from the left and right neighboring image with respect to the reference image, before Symbol and it outputs the pixel values of the near neighbor depth corresponding pixel.

  The depth estimation program according to the ninth aspect of the present invention is a computer for estimating a depth value indicating the depth of a subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position. Are configured to function as video input means, corresponding pixel difference calculation means, neighborhood pixel value output means, neighborhood depth corresponding pixel value output means, difference value determination means, smoothing means, depth value determination means, and neighborhood depth value output means. did.

  In such a configuration, the depth estimation program uses the video input means to capture a reference image obtained by photographing the subject with a predetermined reference camera in any of the plurality of cameras arranged, and the subject with an adjacent camera other than the reference camera. Input a plurality of adjacent images taken of. Then, the depth estimation program calculates the pixel value of the pixel of interest and the assumed depth value of the pixel of interest for each pixel of interest of the assumed depth value and the reference image assuming the depth value by the corresponding pixel difference calculation means. A corresponding pixel difference value that is an absolute difference value with respect to the pixel values of the corresponding pixels of the plurality of adjacent videos at the pixel position corresponding to the parallax is calculated.

  Then, the depth estimation program outputs the pixel value of the target pixel of the reference video and the predetermined number of neighboring pixels from the target pixel to the left and right in the reference video input to the video input unit by the neighboring pixel value output unit. A pixel value of a neighboring pixel at a distant pixel position is output. Further, the depth estimation program responds to the parallax of the depth value of the neighboring pixels from the adjacent images on the left and right with respect to the reference image for each of the assumed depth value and the target pixel by the neighboring depth corresponding pixel value output unit. The pixel value of the neighborhood depth corresponding pixel at the pixel position is output.

  Then, the depth estimation program uses the difference value determination means to calculate an average difference value that is an average value of the corresponding pixel difference values and a minimum value that is a minimum value of the corresponding pixel difference values for each of the assumed depth value and the target pixel. A difference value is obtained, and based on the pixel value of the target pixel, the pixel value of the neighboring pixel, the depth value of the neighboring pixel, and the pixel value of the neighboring depth corresponding pixel, either the minimum difference value or the average difference value Is determined as a selected difference value according to a predetermined condition, and the selected difference value is written in the storage means in association with the assumed depth value and the pixel position of the reference image.

  Then, the depth estimation program causes the smoothing unit to calculate, for each target pixel, an assumed depth value that minimizes a selection difference value of a pixel adjacent to the target pixel and the target pixel's selection difference value. A smoothing evaluation value that is a sum of the smoothed difference values is calculated by calculating a smoothed difference value obtained by smoothing the selected difference value by weighting and adding a difference with an assumed depth value at which the selected difference value is minimized. And the smoothed difference value is written in the storage means in association with the assumed depth value and the pixel position of the reference image.

  Then, the depth estimation program replaces the assumed depth value corresponding to the selection difference value stored in the storage means with the second assumed depth value in which a depth value different from the assumed depth value is assumed by the depth value determining means. When the recalculated evaluation value corresponding to the smoothed evaluation value is calculated and the recalculated evaluation value is less than the smoothed evaluation value calculated by the smoothing means, the second hypothetical depth value is calculated from the reference image. If the recalculated evaluation value is not less than the smoothed evaluation value, the assumed depth value corresponding to the selection difference value stored in the storage means is determined as the depth value of each pixel of the reference image. Determine as. Further, the depth estimation program receives the depth value of each pixel of the reference image from the depth value determination means by the neighborhood depth value output means, and outputs the depth value of the neighboring pixels to the neighborhood depth corresponding pixel value output means and Output to the difference value determining means.

The present invention has the following excellent effects.
According to the first, eighth, and ninth inventions of the present application, even when some subjects with different colors have occlusion, the color of the subject is determined by comparing the pixel value of the target pixel with the pixel value of the neighboring depth corresponding pixel. Whether or not there is occlusion regardless of the difference. When it is determined that there is occlusion, according to the first, eighth, and ninth inventions of the present application, the minimum difference value, that is, the difference value on the side without occlusion can be correctly selected as the selection difference value.

  Further, according to the first, eighth, and ninth inventions of the present application, even when false matching occurs due to a subject having a uniform texture, the pixel value of the target pixel is compared with the pixel value of the neighboring depth corresponding pixel. Determine if there is occlusion. When it is determined that there is no occlusion, according to the first, eighth, and ninth inventions, the average difference value is selected as the selected difference value in order to prevent false matching. Thus, since a correct selection difference value can be selected, according to the first, eighth, and ninth inventions of the present application, it is possible to reduce erroneous estimation of the depth value.

  According to the second invention of this application, when the target pixel is not near the boundary of the subject and there is no occlusion, false matching is prevented by selecting the average difference value as the selection difference value. Thus, according to the second invention of the present application, a more correct selection difference value can be selected, and the depth value estimation accuracy can be further improved.

  According to the third invention of this application, when there is occlusion in the left and right corresponding pixels, the selection difference value on the side without occlusion is selected from the left and right difference absolute values. Thus, according to the third invention of the present application, a correct selection difference value can be selected, and the depth value estimation accuracy can be further improved.

  According to the fourth aspect of the present invention, even if the target pixel is near the boundary of the subject, if there is no occlusion in the adjacent video, false matching is prevented by selecting the average difference value as the selection difference value. Thus, according to the third invention of the present application, a correct selection difference value can be selected, and the depth value estimation accuracy can be further improved.

  According to the fifth aspect of the present invention, in the first depth estimation, by selecting an average difference value, false matching can be prevented, and a correct depth value can be estimated even if the subject has a uniform texture. Furthermore, according to the fifth invention of the present application, by using the estimated depth value, the depth estimation is performed for the second and subsequent times, so that the presence or absence of occlusion can be accurately determined, and the estimation accuracy of the depth value is further improved. Can be made.

  According to the sixth aspect of the present invention, even when the target pixel is located at the end of the reference image and the corresponding pixel is deviated from the left and right adjacent images, another corresponding image in which the corresponding pixel is not deviated is used. The absolute difference value of is calculated. Thus, according to the sixth aspect of the present invention, false matching can be prevented, and the depth value estimation accuracy can be further improved.

  According to the seventh invention of the present application, even if the subject has a very large parallax, the pixel value of the neighboring depth corresponding pixel at the pixel position where there is a possibility of occlusion in the adjacent video is checked. Can be judged. Further, according to the seventh invention of the present application, even if the subject is narrow, the presence / absence of occlusion is determined by changing the distance to the adjacent pixel, so this subject is rarely missed. As described above, according to the seventh invention of the present application, it is possible to correctly determine the presence or absence of occlusion, to miss an object having a very large parallax, and to further improve the estimation accuracy of the depth value.

It is a block diagram which shows the structure of the depth estimation system which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structure of the depth estimation apparatus which concerns on 1st Embodiment of this invention. In the present invention, when there is occlusion but the assumed depth value is correct, it is an explanatory diagram for explaining the relationship between a pixel of interest and left and right corresponding pixels. In this invention, it is explanatory drawing for demonstrating the case where a corresponding pixel remove | deviates from the left and right adjacent image | video. In this invention, it is explanatory drawing for demonstrating the relationship between an attention pixel and a neighboring pixel. In this invention, it is explanatory drawing for demonstrating the depth value of a neighboring pixel. In this invention, it is explanatory drawing for demonstrating the neighborhood depth corresponding | compatible pixel corresponding to the depth value of a neighborhood pixel. It is a flowchart which shows the whole operation | movement of the depth estimation apparatus of FIG. It is a flowchart which shows the corresponding pixel difference calculation operation of the depth estimation apparatus of FIG. It is a flowchart which shows the difference value determination operation | movement of the depth estimation apparatus of FIG. It is a flowchart which shows the neighborhood depth value output operation | movement of the depth estimation apparatus of FIG. It is a flowchart which shows the difference value selection operation | movement of the depth estimation apparatus of FIG. It is a flowchart which shows the neighborhood pixel value output operation | movement of the depth estimation apparatus of FIG. It is a flowchart which shows the neighborhood depth corresponding | compatible pixel value output operation | movement of the depth estimation apparatus of FIG. It is a flowchart which shows the smoothing operation | movement of the depth estimation apparatus of FIG. It is a flowchart which shows the depth value determination operation | movement of the depth estimation apparatus of FIG. It is a block diagram which shows the structure of the depth estimation system which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the depth estimation apparatus which concerns on 2nd Embodiment of this invention. FIG. 19 is a flowchart illustrating a corresponding pixel difference calculation operation of the depth estimation device in FIG. 18. FIG. It is explanatory drawing for demonstrating the conventional depth estimation method.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, means having the same function are denoted by the same reference numerals and description thereof is omitted.

(First embodiment)
[Overall configuration of depth estimation system]
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.
With reference to FIG. 1, the whole structure of the depth estimation system S containing the depth estimation apparatus 1 which concerns on 1st Embodiment of this invention is demonstrated. Depth estimation system S from the multi-view images obtained by photographing the subject T in several sequences the camera CM, and generates a depth image F _Z estimating the subject T depth value D (see FIG. 2). The depth estimation system S shown in FIG. 1 includes a camera CM (C _LL , C _L , C _C , C _R , C _RR ) and a depth estimation device 1. A plurality of cameras C _LL , C _L , C _C , C _R , and C _RR are collectively referred to as a camera CM.

The camera CM is a general photographing device that photographs the subject T. Here, it is possible to generate a depth value D in accordance with the horizontal parallax with respect to serving as a reference camera (reference camera) C _C parallax, and spaced from each other at regular intervals by a distance L determined in advance in the horizontal direction The cameras (adjacent cameras) C _LL , C _L , C _R , and C _RR are arranged in parallel. Videos (F _LL , F _L , F _C , F _R , F _RR ) captured by this camera CM (C _LL , C _L , C _C , C _R , C _RR ) are input to the depth estimation device 1. .
The distance L is arbitrary as long as the depth value D and the amount of parallax can be accommodated. The subject T includes a background A and a foreground B, for example.

The camera C _C is for shooting the image (reference image F _C) as a reference of parallax. The cameras C _L and C _R are cameras arranged on the left and right sides of the camera C _C , respectively, and shoot images (adjacent images F _L and F _R ) for obtaining parallax with the reference image F _C. It is. Further, the camera C _RR is a camera C _R right arranged camera is for capturing a neighboring image F _RR to determine the corresponding pixels on behalf of the neighboring image F _L. Further, the camera C _LL is a camera arranged on the left of the camera C _L, is intended to take a neighboring image F _LL for determining the corresponding pixel in place of the neighboring image F _R.

If the depth value D is generated according to the vertical parallax, the camera CMs may be arranged apart from each other in the vertical direction.
In addition, the video (F _LL , F _L , F _C , F _R , F _RR ) captured by the camera CM may be a still image or a moving image. In the case of a moving image, the camera CM performs frame synchronization with each other, and sequentially inputs the frame image of the still image to the depth estimation device 1. Alternatively, a time code may be added for each frame image, and the depth estimation apparatus 1 may be synchronized.

The depth estimation apparatus 1 estimates a depth value indicating the depth of the subject T from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged at predetermined positions. Here, the depth estimation apparatus 1 includes a camera _CM arranged in the horizontal direction _{(C LL, C L, C} C, C R, C RR) camera video _{(F LL} obtained by photographing a subject T in, F _{L, F} C, Depth value D indicating the depth of subject T is estimated from F _R , F _RR ). Here, the depth estimation apparatus 1, the depth value D, and the generating a depth image F _Z by associating to each pixel value of the image.

[Configuration of depth estimation device]
With reference to FIG. 2, the structure of the depth estimation apparatus 1 which concerns on 1st Embodiment of this invention is demonstrated (refer FIG. 1 suitably). As shown in FIG. 2, the depth estimation apparatus 1 includes a video input unit 10, a corresponding pixel difference calculation unit 20, a neighboring pixel value output unit 30, a neighboring depth value output unit 40, and a neighboring depth corresponding pixel value output unit. 50, a difference value determining unit 60, a storage unit 70, a smoothing unit 80, and a depth value determining unit 90.

The video input means 10 inputs a plurality of camera videos taken by the camera CM. Here, the image input means 10, the reference image _{F C} taken by the base camera _{C C} a predetermined, adjacent camera _{C LL, C L, C R} , neighboring image _F LL shot by _{C RR, F} L, F Input _{R 1} and F _RR .
Each video (F _LL , F _L , F _C , F _R , F _RR ) input by this video input means 10 is stored in a memory not shown in the figure, and corresponding pixel difference calculation means 20, which will be described later, and neighboring pixel value output Reference is made to the means 30 and the neighborhood depth corresponding pixel value output means 50.

Here, the reference image F _C and the left and right neighboring image F _L when occlusion to the pixel of interest P _C is _present, the appearance of F _R supplement the description. As shown in FIG. 3, the pixel of interest P _C, who is on the background A close to the right side of the foreground B, the reference image F _C and the neighboring image F _L In point of interest PC is visible. Meanwhile, the neighboring image F _R, the point of interest PC is not hidden in the foreground B, the corresponding point PR on the foreground B is visible instead. Thus, by selecting the left of the corresponding pixel corresponding pixel difference values between P _L and the target pixel P _C a (matching error), you can get the correct corresponding pixel difference values. Meanwhile, if you choose the corresponding pixel difference value of the right of the corresponding pixel P _R, or, if you choose the average of the left and right of the corresponding pixel difference values, would get the wrong corresponding pixel difference value, the depth value Is erroneously estimated. Therefore, the depth estimation apparatus 1 examines the depth value and the pixel value of the right and left near the target pixel P _C, by determining the presence or absence of occlusion to the pixel of interest P _C, the correct choice of matching error.

Hereinafter, returning to FIG. 2, the description of the configuration of the depth estimation apparatus will be continued (see FIG. 3 as appropriate).
Corresponding pixel difference calculating unit 20, for each hypothetical depth value d (see FIG. 3) and the target pixel P _C, the pixel value of the pixel of interest P _C of the reference image F _C, assumptions depth value d of the pixel of interest P _C which arithmetic neighboring image _F L corresponding to the parallax, the corresponding pixel _P L of _{F R, P R} corresponding pixel difference value _E CL of neighboring image which is a difference absolute value between the pixel value (see FIG. _3), the _{E CR} It is. Here, the corresponding pixel difference value E _CL of neighboring image are calculated difference absolute value between the reference image F _C and the neighboring image F _L, the corresponding pixel difference value E _CR of the neighboring image, the reference image F _C to be the calculated difference absolute value between the neighboring image F _R.

Note that the absolute difference value of the pixel value is the luminance of the pixel when the camera image (F _L , F _C , F _R ) is composed of each element of luminance (luminance value) and color difference signal (color difference value). A value obtained by averaging the difference absolute value of the value and the difference absolute value of the color difference value is obtained. At this time, the color component weight may be adjusted by multiplying the absolute difference value of the color difference value by an arbitrary coefficient.
In addition, when the camera image (F _L , F _C , F _R ) is composed of RGB color signals, the absolute difference value of the pixel value is the difference absolute value of the R value, which is an element of the color signal, and the G value. The difference absolute value of B and the difference absolute value of the B value are added and averaged.

Also, assuming the depth value d is a value for temporarily set the depth value D of the subject T, taking a value from the minimum amount of parallax to a depth value D corresponding to the maximum disparity amount D _M. For example, when the depth value D is represented by 8-bit data, the assumed depth value d takes a value ranging from “0” (minimum parallax amount) to “255” (maximum parallax amount). Here, the corresponding pixel difference calculation means 20 sequentially sets the assumed depth value d from the minimum parallax amount (for example, “0”) to the depth value D corresponding to the maximum parallax amount (for example, “255”). Then, the difference between the pixel values corresponding to the corresponding pixels corresponding to the parallax of the assumed depth value d is calculated as the corresponding pixel difference value. Here, the unit of the depth value D and the unit of the parallax between the pixels are the same, and when the depth value D increases by “1”, the parallax increases by one pixel.

Corresponding pixel difference calculating means 20, the reference image _{F C} (x, y) coordinates (the pixel of interest _{P C)} the pixel value at _P C (x, y), the reference image _{F C} (x, y) assumes the coordinates depth value (x + d, y) of the neighboring image _{F L} corresponding to d coordinates (corresponding pixels _{P L)} a pixel value of _{P L (x + d, y} ), (x-d, y) of the neighboring image _{F R} coordinate (corresponding When the pixel value of the pixel P _R ) is P _R (x−d, y), the corresponding pixel difference values E _CL (d, x, y), E _CR (d, x, y) are expressed by the following equation (1). y) is calculated. “||” indicates an absolute value.
E _CL (d, x, y) = | P _L (x + d, y) −P _C (x, y) |
E _CR (d, x, y) = | P _R (x−d, y) −P _C (x, y) | (1)

Here, with reference to FIG. 4, if the corresponding pixels _{P L} from neighboring image _{F L} is out, the corresponding pixel difference value _{E CL,} method of calculating _{E CR} illustrating (appropriately with reference to Figure 2). As shown in FIG. 4, when the neighboring image F _L corresponding pixel P _L deviates not Motomara the left of the corresponding pixel difference value E _CL, using only the right of the corresponding pixel difference value E _CR, the depth The value must be estimated. At this time, when the object T present in the vicinity of the pixel of interest P _C is uniform texture, the corresponding pixel difference value E _CR is becomes a value approximating in any hypothetical depth value d, it can not estimate the correct depth value . In this case, with respect to the reference image F _C , in the adjacent images F _R and F _RR on the right side opposite to the adjacent image F _L on the side where the corresponding pixel P _L is removed, the corresponding pixels P _R and P _RR must be removed. Conceivable. Therefore, the corresponding pixel difference calculation means 20 calculates the corresponding pixel difference values E _CL and E _CR using the corresponding pixels P _R and P _RR of the adjacent images F _R and F _RR .

First, the corresponding pixel difference calculating unit 20 determines the neighboring image _F L, _F the corresponding pixel from the _{R P} L, whether or not disconnected _{P R.} Specifically, the corresponding pixel difference calculating means 20, the pixel positions of the corresponding pixels P _R is moved to the left x-coordinate value of the pixel of interest P _C by assuming the depth value d (x-d) is, is 0 or more It is determined whether or not. That is, the corresponding pixel difference calculating means 20, by the following equation (2), the corresponding pixel P _R is determined whether not out from neighboring image F _R.
x−d ≧ 0 (2) formula

Further, the corresponding pixel difference calculating means 20, the pixel position of the moved to the right x-coordinate value of the pixel of interest P _C by assuming the depth value d corresponding pixel P _L (x + d) is preset picture size (e.g., 1920 It is determined whether it is less than (pixel). That is, the corresponding pixel difference calculating means 20, by the following equation (3), the corresponding pixel P _L is determined whether not out from neighboring image F _L.
x + d <video width (3) formula

In the example of FIG. 4, since the corresponding pixel P _L from neighboring image F _L is out, does not satisfy the equation (3). In this case, the corresponding pixel difference calculation means 20 calculates the corresponding pixel difference values E _CL and E _CR by the following equation (4).
E _CL (d, x, y) = | P _RR (x−2d, y) −P _C (x, y) |
E _CR (d, x, y) = | P _R (x−d, y) −P _C (x, y) | (4)

Not shown, but if the right neighboring image F _R corresponding pixel P _R deviates not satisfy the equation (2). In this case, the corresponding pixel difference calculation means 20 calculates the corresponding pixel difference values E _CL and E _CR by the following equation (5).
E _CL (d, x, y) = | P _L (x + d, y) −P _C (x, y) |
E _CR (d, x, y) = | P _LL (x + 2d, y) −P _C (x, y) | (5)

Further, the corresponding pixel difference calculating unit 20, neighboring image _F L, the corresponding pixel _P L from _{F R,} if not disconnected _{P R,} i.e., if it meets both the (2) and (3), (1) Corresponding pixel difference values E _CL and E _CR are calculated by the equation. Thereby, the depth estimation apparatus 1, neighboring image _F L, _{F R} from the corresponding pixels _P L, even if _{the P R} is outside, two adjacent image _F R on the opposite _side, F _RR, F _L, with _{F LL} , The corresponding pixel difference values E _CL and E _CR can be calculated, and erroneous estimation of the depth value can be reduced.

Then, the corresponding pixel difference calculation unit 20 outputs the corresponding pixel difference values E _CL and E _CR for each assumed depth value d (minimum parallax amount to maximum parallax amount) to the difference value determination unit 60. The correspondence pixel difference calculating means 20, the pixel position of the pixel of interest P _C to be calculated (x, y) and sequentially updating the assumed depth value d, the corresponding pixel difference value E _CL, calculates the E _CR.

Hereinafter, the neighboring pixel value output means 30 will be described with reference to FIG. 5 (see FIG. 2 as appropriate).
Neighboring pixel value output unit 30 receives the reference video _{F C} from the image input unit 10, by the following equation (6), the pixel value _{P C} of the pixel of interest _P C (x, y), the pixel of interest _P C ( The pixel value P _SL of the neighboring pixel located at the pixel position P _C (x−s, y) left from the neighboring pixel number s from x, y) and the neighboring pixel number s from the target pixel P _C (x, y) pixel position _{P C (x + s, y} ) apart right by obtaining the pixel value _{P SR} neighboring pixels in the.
P _C = P _C (x, y)
_{P SL = P C (x-} s, y)
_{P SR = P C (x +} s, y) ... (6) formula

Then, the neighboring pixel value output means 30, the pixel value _{P C} of the pixel of interest _P C (x, y), the pixel value _{P SL} neighboring pixels in the pixel position _{P C (x-s, y} ), the pixel position _{P C (x + s, y} ) and the pixel value _{P SR} neighboring pixels in the outputs to the difference value determination means 60.

Note that the pixel of interest P _C, which is a pixel of interest to estimate the depth value D in the reference image F _C.
Further, the neighboring pixel P _SL and its neighboring pixels P _SR, as shown in FIG. 5, around the pixel of interest P _C, is a pixel radius positioned on the circumference of the neighboring pixel number s. The neighboring pixel number s is, for example, to correctly determine whether the occlusion is to set the number of pixels corresponding to the maximum disparity amount D _M as the initial value. In the following drawings, the left neighboring pixel may be illustrated as the left neighboring pixel, and the right neighboring pixel may be illustrated as the right neighboring pixel.

Hereinafter, the neighborhood depth value output means 40 will be described with reference to FIG. (See FIG. 2 as appropriate).
Near depth value output means 40, described later to the depth value determining means 90 from the respective pixel depth value D (x, y) is inputted, following the expression (7), by the number of neighboring pixels s from the target pixel P _C Left depth value _D of the neighboring pixels _{P SL} of spaced L = D (x-s, y) and, the target pixel _{P C} neighboring pixels _{P SR} apart right by neighboring pixel number s from the depth value D = _{D R} (x + s , Y). In the following figures, the depth value of the neighboring pixel is illustrated as the neighborhood depth value, the depth value of the left neighboring pixel is illustrated as the left neighborhood depth value, and the depth value of the right neighboring pixel is represented as the right neighborhood depth value. May be illustrated.
D _L = D (x−s, y)
D _R = D (x + s, y) (7)

Also, near the depth value output means 40, by the following equation (8), a difference absolute value between the left depth value _{D L} and the right depth value _{D R |} or is greater than a predetermined threshold value th1 | D L -D _R Determine whether or not. The threshold th1 is a value obtained by multiplying a value obtained by subtracting the minimum parallax amount (eg, “0”) from the maximum parallax amount D _M (eg, “255”) by a predetermined coefficient (eg, “0.2”). Are set in advance.
| D _L −D _R |> th1 (8)

When the difference absolute value | D _L −D _R | does not exceed the threshold th1, the neighborhood depth value output means 40 divides the number of neighboring pixels s by half as in the following equation (9), and the above equation (7). Thus, the depth value D _L = D (x−s, y) of the neighboring pixel P _SL and the depth value D _R = D (x + s, y) of the neighboring pixel P _SR are acquired again. In equation (9), “=” means that the value on the left side is replaced with the value on the right side.
s = s / 2 (9) formula

Then, the neighborhood depth value output means 40 obtains the depth value D _L = D (x−s, y) of the acquired neighborhood pixel P _SL and the depth value D _R = D (x + s, y) of the neighborhood pixel P _SR. And output to the neighborhood depth corresponding pixel value output means 50 and the difference value determination means 60. Thus, by obtaining the depth value of the pixel position apart maximum disparity amount D _M, the depth estimation apparatus 1, neighboring image F _L, to correctly determine the foreground A that may occlusion is at a F _R Can do. That is, the difference value determination means 60, since using the maximum parallax amount D _M as the initial values of the neighboring pixel number s, never miss a subject T that may occlusion there. Furthermore, the depth estimation apparatus 1 can determine the presence or absence of occlusion even in the foreground A having a narrow width, which has been difficult to determine, by determining the depth value again by halving the number of neighboring pixels s. .

Note that the depth value D cannot be obtained in the first depth estimation. In this case, the depth estimation apparatus 1 sets the depth value D of all the pixels in advance to an initial value (for example, “0”). That is, when performing the first depth estimation, near the depth value output means 40 outputs the initial value depth value D _L, in the vicinity of the depth corresponding pixel value output unit 50 and difference value determination unit 60 as D _R. On the other hand, in the second and subsequent depth estimation, the neighboring depth value output means 40 receives the depth value D of each pixel obtained in the previous depth estimation from the depth value determination means 90, so that the depth of the correct neighboring pixel _PSL is correct. can output the value _{D L,} the right neighboring pixels _{P SR} and depth value _{D R.}

  Hereinafter, the neighborhood depth corresponding pixel value output means 50 will be described with reference to FIG. 7 (see FIG. 2 as appropriate). In the following drawings, the left neighboring depth corresponding pixel may be illustrated as the left neighboring depth corresponding pixel, and the right neighboring depth corresponding pixel may be illustrated as the right depth corresponding pixel.

Neighboring depth corresponding pixel value output unit 50, image input unit 10 from the reference image _{F C} and the neighboring image _F L, _{F R} is input, the depth values _{D L} and its neighboring pixels of neighboring pixels _{P SL} from neighboring depth value output means 40 depth value _{D R} of P _SR is input. Then, the neighboring depth corresponding pixel value output means 50 calculates the neighboring depth corresponding pixel P _{L at} the pixel position corresponding to the parallax of the depth value D _L from the left and right adjacent images F _L and F _R by the following equation (10). Pixel values P _LDL and P _RDL of (x + D _L , y) and P _R (x−D _L , y) are calculated. Also, near the depth corresponding pixel value output unit 50, by (10), left and right neighboring image _F L, from _{F R,} the depth value _D near depth corresponding pixel is in the pixel position corresponding to parallax of _{R P L} (x + D _{R, y), P R (} x-D R, y) pixel value _P LDR _of calculating the _{P RDR.}
P _LDL = P _L (x + D _L , y)
P _LDR = P _L (x + D _R , y)
P _RDL = P _R (x−D _L , y)
P _RDR = P _R (x−D _R , y) (10)

Then, the neighboring depth corresponding pixel value output means 50 includes the neighboring depth corresponding pixels P _L (x + D _L , y), P _R (x−D _L , y), and the neighboring depth corresponding pixels P _LDL , P _{RDL. L (x + D R, y} ), and outputs _{P R (x-D R,} y) pixel value _P LDR _of the _{P RDR} to the difference value determination means 60.

Hereinafter, returning to FIG. 2, the description of the configuration of the depth estimation device 1 will be continued.
Difference determining means 60, for each pixel of interest P _C and assumptions depth value d of the reference image F _C, and requests the minimum difference value Em and the average difference value Ea. And the difference value determination means 60 determines either the calculated | required minimum difference value Em or the average difference value Ea as a selection difference value by the preset conditions.

  Here, the difference value determining means 60 includes an average difference value calculating means 60a, a minimum difference value selecting means 60b, and a difference value selecting means 60c.

Average differential value calculating means 60a, for each pixel of interest P _C and assumptions depth value d of the reference image F _C, the corresponding pixel difference value E _CL input from the corresponding pixel difference calculating unit _20, the average value of E _CR (average difference The value Ea) is calculated.
That is, the average difference value calculating means 60a calculates the average difference value Ea (d, x, y) corresponding to the assumed depth value d of the (x, y) coordinates of the reference image F _C according to the following (11). .
Ea (d, x, y) = {E _CL (d, x, y) + E _CR (d, x, y)} / 2
... (11) Formula

  Then, the average difference value calculation means 60a outputs the average difference value Ea for each assumed depth value d (“0” to the maximum parallax amount) obtained by the calculation to the difference value selection means 60c.

Minimum difference value selecting unit 60b, for each pixel of interest P _C and assumptions depth value d of the reference image F _C, corresponding pixel difference calculating unit 20 corresponding pixel difference value E _CL input _from, among the E _CR, the value The smaller one (minimum difference value Em) is selected.
That is, the minimum difference value selection unit 60b uses the following equation (12) to calculate the corresponding pixel difference value E _CL (d, x) of the adjacent image corresponding to the assumed depth value d of the (x, y) coordinates of the reference image F _C. , Y) and the corresponding pixel difference value E _CR (d, x, y) of the adjacent video are compared, and the smaller value is selected as the minimum difference value Em. In equation (12), min is a function that selects the smaller one of the corresponding pixel difference values E _CL and E _CR described in “{}”.
Em (d, x, y) = min {E _CL (d, x, y), E _CR (d, x, y)}
... (12) Formula

  Then, the minimum difference value selecting unit 60b outputs the minimum difference value Em for each selected assumed depth value d (“0” to the maximum parallax amount) to the difference value selecting unit 60c.

Difference value selection unit 60c, for each pixel of interest _{P C} and assumptions depth value d of the reference image _{F C,} the pixel value _{P C} of the target pixel is input from the neighboring pixel value output means 30 _P C (x, y), and the pixel value _{P SL} neighboring pixels in the pixel position _{P C (x-s, y} ) , the pixel value _{P SR} neighboring pixels in the pixel position _{P C (x + s, y} ), input from the vicinity depth value output means 40 a depth value _{D L} neighboring pixels _{P SL} that is, a depth value _{D R} neighboring pixels _{P SR,} the pixel value _P LDL neighboring depth corresponding pixel that is input from the vicinity depth corresponding pixel value output unit _{50, P RDL,} P _LDR, based on the P _RDR selected minimum difference value Em inputted from the minimum difference value selecting means 60b or one of the average difference value Ea input from the average difference value calculation means 60a,, as the selected difference value E To do.

First, the difference value selection unit 60c determines whether the depth estimation number n indicating the number of times the depth value D is estimated is the first time or the second time or later. When the depth estimation number n is the first (n = 1), the difference value selection means 60c calculates the average difference value Ea (d, x, y) as the selected difference value E (d, x) by the following equation (13). , Y).
E (d, x, y) = Ea (d, x, y) (13)

If depth estimation the number n is the second and subsequent (n ≧ 2), the difference value selecting unit 60c, the following equation (14) by the depth value _D L of the right and left neighboring pixels, the difference absolute value of _{D R} | _{D L} -D _R | exceeds the threshold value th1, and the left and right neighboring pixels _{P SL,} determines whether or not the difference absolute value _{E LR} of the pixel values of _{P SR} exceeds the threshold value th2. The threshold th2 is set in advance as a value obtained by multiplying the maximum pixel value (for example, “255”) by a predetermined coefficient (for example, “0.1”).
| D _L -D _R |> th1 AND
E _LR = | P _SL −P _SR |> th2 (14)

If (14) is not satisfied, the boundary of the object T to the right and left near the target pixel P _C (i.e., it is most likely not a boundary between the background A foreground B. In this case, the difference value selecting unit 60c, the The average difference value Em (d, x, y) is selected as the selected difference value E (d, x, y) by the equation (13), and the selected difference value E (d, x, y) is stored in the storage means 70. Write to.

(14) If the equation is satisfied, it is highly likely that the depth value and the pixel value and differ object T on the left and right near the target pixel P _C is present, it is necessary to check for occlusion. In this case, the difference value selecting unit 60c, the following by equation (15), the depth value D _L is the depth value D by determining whether a _R or a background A the depth value decreases in the left or right side Determine if there is any.
D _L ≧ D _R (15) formula

(15) If the equation is satisfied (if the depth value D _L is greater than the depth value D _R), there is a foreground B to the left of the pixel of interest P _C, it is likely that there is a background A on the right. In this case, the difference value selecting unit 60c determines the pixel of interest P _C Do is on or foreground B is the background A. Here, if the pixel of interest P _C background A, there is occlusion in the left neighboring image F _L, the pixel value of the pixel values between the target pixel P _C of the corresponding pixel P _L is greatly different. Meanwhile, occlusion in the right side of the neighboring image F _R is not, the pixel value and the pixel value and is close to the value of the target pixel P _C of the corresponding pixel P _R. Therefore moving difference value selecting unit 60c is the following equation (16), left and right neighboring image _F L, in _{F R,} the left and right by the number of pixels corresponding to the pixel of interest _{P C} to a depth value _{D R} Background A side pixel value _P LDR neighboring depth corresponding pixels is _calculated and _{P RDR,} differential absolute value _E LDR between the pixel value _{P C} of the pixel of _interest, the _{E RDR.
E LDR = | P LDR -P C} |
E _RDR = | P _RDR −P _C | (16)

Then, the difference value selection means 60c determines whether or not the difference absolute value _ERDR exceeds the difference absolute value _ERDR by the following equation (17).
E _LDR > E _RDR (17) formula

If (17) is established (when the difference absolute value _{E LDR} exceeds the difference absolute value _{E RDR),} it can be said that occlusion is present in the left neighboring image _{F L.} In this case, the difference value selection means 60c selects the minimum difference value Em (d, x, y) as the selected difference value E (d, x, y) by the following equation (18).
E (d, x, y) = Em (d, x, y) (18)

Thus, the pixel of interest P _C is assumed to be at the same depth as the background A, by examining the left and right neighboring image F _L, the appearance of the pixel of interest P _C in F _R, determines the presence or absence of occlusion. At this time, the depth estimation apparatus 1, since used to determine the proximity depth corresponding pixel P _LDR, P _RDR corresponding to the pixel of interest P _C, even if some of the colors are different subject T, it can be determined whether the correct occlusion . Further, the depth estimation apparatus 1, the corresponding pixel P _L corresponding to the assumed depth value _d, is not used to determine the P _R, even when assuming the depth value d is not correct, it can be determined whether the correct occlusion.

When the equation (17) is not satisfied (when the difference absolute value _ERDR does not exceed the difference absolute value _ERDR ), it can be said that there is no occlusion. In this case, the difference value selection means 60c selects the average difference value Ea (d, x, y) as the selected difference value E (d, x, y) according to the equation (13) in order to prevent false matching. .

If (15) is not satisfied (if the depth value D _L is less than the depth value D _R), there is a background A on the left side of the target pixel P _C, there will be foreground B on the right. In this case, the difference value selecting unit 60c is the following equation (19), left and right neighboring image _F L, in _{F R,} the left and right by the number of pixels corresponding to the pixel of interest _{P C} to a depth value _{D L} background A side pixel value _P LDL neighboring depth corresponding pixel that has _moved, computes a _{P RDL,} differential absolute value _E LDL between the pixel value _{P C} of the pixel of _interest, the _{E RDL.
E LDL = | P LDL -P C} |
_{E RDL = | P RDL -P C} | ... (19) equation

Then, the difference value selection unit 60c determines whether or not the difference absolute value E _RDL exceeds the difference absolute value E _LDL by the following equation (20).
E _LDL <E _RDL (20) formula

(When the difference absolute value _{E RDL} exceeds the difference absolute value _{E LDL)} if (20) is established, it can be said that occlusion is present in the right neighboring image _{F R.} In this case, the difference value selection means 60c selects the minimum difference value Em (d, x, y) as the selected difference value E (d, x, y) by the above-described equation (18).

When the equation (20) is not satisfied (when the difference absolute value E _RDL does not exceed the difference absolute value E _LDL ), it can be said that there is no occlusion. In this case, the difference value selection means 60c selects the average difference value Ea (d, x, y) as the selected difference value E (d, x, y) by the above-described equation (13) in order to prevent false matching. To do.

Then, the difference value determination unit 60 writes in the storage means 70 elected difference value E (d, x, y) and the pixel position of the hypothetical depth value d and the reference image F _C (x, y) in association with . Also, the difference value determination means 60, at all pixel positions of the reference image F _C, selected difference value E for each hypothetical depth value d (d, x, y) after writing in the storage means 70, smoothing the write completion Notification to the generating means 80.

  As described above, the difference value determining means 60 prevents false matching when the assumed depth value d is incorrect by always selecting the average difference value Ea as the selected difference value E in the first depth estimation. Can do.

Since only the average difference value Ea is selected in the first depth estimation, there is a possibility that the difference value determining means 60 erroneously estimates the depth value in a portion where there is occlusion. Therefore, in the second and subsequent depth estimation, the difference value determination means 60, neighboring pixels are estimated at a position away from the portion where the occlusion is present P _SL, using a depth value D _L, D _R of P _SR, occlusion Determine the presence or absence correctly. Further, the difference value determination means 60 corrects the first estimation of the depth value by selecting the minimum difference value Em as the selection difference value E in a portion where there is occlusion. Thereby, the depth estimation apparatus 1 can increase the probability that the correct depth value D is estimated in the depth value determination means 90 described later.

Furthermore, the difference value determination means 60, whether the pixel of interest _{P C} is either a background A is foreground B, neighboring pixel _{P SL,} without determination by the pixel values of _{P SR,} left and right neighboring image _F L, F _{In R} , determination is made based on neighboring depth corresponding pixels P _LDR , P _RDR , P _LDL , and P _RDL . Probability Thus, the depth estimation apparatus 1, even if a part of the color of different objects T, since the pixel of interest P _C can determine whether it is foreground B or background A, the correct depth value D is estimated Can be high.

  The storage means 70 stores the selected difference value E (d, x, y) selected by the difference value determination means 60 in association with the assumed depth value d and the pixel position (x, y) of the reference image. is there. Further, the storage unit 70 associates the smoothed difference value Ev (d, x, y) smoothed by the smoothing unit 80 described later with the assumed depth value d and the pixel position (x, y) of the reference image. Remember. For example, the storage means 70 is a general storage medium such as a magnetic memory or a semiconductor memory.

The smoothing means 80 smoothes the selection difference value at the pixel position. First, the smoothing unit 80, the pixel of interest _{P C} (pixel position (x, y)) of the reference image _{F C} per, by the following equation (21), the pixel of interest _{P C} of the selected difference value E (d, x , Y) is calculated assumable depth value d that minimizes. Then, the smoothing unit 80 sets the assumed depth value d as the depth value D _{x, y of the} pixel.
D _{x, y} = D (x, y) = min {E (d, x, y)} (21)

As shown in the equation (21), the selection difference value E (d, x, y) is a function of the pixel position (x, y) and the assumed depth value d. Therefore, min is a function that selects and returns an assumed depth value d that minimizes the selected difference value E (d, x, y) for each pixel position (x, y). The return value D (x, y) of this function min is simply described as D _{x, y} .

Next, the smoothing unit 80, according to the above mentioned (21), similarly to the pixel of interest _{P C,} pixel position adjacent to the pixel of interest _{P C (x + 1, y} ), select the difference between adjacent pixels (x, y + 1) Assumed depth values D _{x + 1, y} and D _{x, y + 1} having the smallest values are calculated.

Then, the smoothing unit 80, by (22), the pixel of interest _{P C} of the selected difference value _{E (D x, y, x} , y) , the assumption depth value _{D x + 1, y, D} x, y + 1 and the depth value D _x, by weighted addition of the difference between _y, and calculates selected difference value _{E (D x, y, x} , y) smoothing the smoothed difference value _{Ev (D x, y, x} , y) and .
Ev (Dx _{, y} , x, y) = E ( _{Dx, y} , x, y)
+ W {| _{Dx + 1, y−} Dx _{, y} | + | _{Dx, y + 1−} Dx _{, y} |} (22)

  In the equation (22), the smoothing coefficient W is manually set to an arbitrary value (for example, “2”). At this time, as the value of the smoothing coefficient W is increased, the selection difference value E becomes smoother, but the possibility of erroneously estimating the depth value of the boundary portion of the subject T increases. For this reason, it is preferable to set the smoothing coefficient W to an appropriate value. Furthermore, the smoothing coefficient W may be input from the outside via an input unit (not shown).

Then, the smoothing unit 80, by the following equation (23), the sum the pixel position (x, y) smoothed difference value _Ev for each _{(D x,} y, x, y) of all pixels of the reference image _{F C} By doing so, the smoothing evaluation value Ev is calculated.
Ev = Σx _{, y} Ev (Dx _{, y} , x, y) (23)

Thereafter, the smoothing unit 80 is smoothed smoothed difference value _{Ev (D x, y, x} , y) and, in association with the pixel positions of the hypothetical depth value d and the reference image _{F C} (x, y) stores Write to means 70. Further, the smoothing means 80 writes the smoothed difference value Ev (D _{x, y} , x, y) for each assumed depth value d in the storage means 70 at all pixel positions of the reference image F _C , The depth value determining means 90 is notified of completion of conversion.

As described above, the smoothing unit 80 smoothes the selection difference value E (D _{x, y} , x, y) at the pixel position (x, y), and smoothes the difference value Ev (D _{x, y} , x, y) can be calculated. As a result, the depth estimation apparatus 1 may have different depth values D _{x, y} that minimize the selection difference value E (D _{x, y} , x, y) between adjacent pixels due to occlusion or false matching. Even so, a smooth depth value can be estimated.

Note that the smoothing means 80, the difference value determination means 60, at all pixel positions of the reference image F _C, by being notified of all assumed depth value indicating that writing the selected difference value d (write complete) Start operation.

Depth value determining means 90, by the following equation (24) and (25), the second assumption depth value D '(hereinafter, the depth value D') while changing the up parallax from the minimum amount of parallax reference picture F _C The recalculated evaluation value Ev ′ for the selection difference value E (D ′, u, v) stored in the storage unit 70 in association with the pixel position (u, v) for each pixel position (u, v) of Is calculated. The recalculated evaluation value Ev ′ corresponds to a smoothing evaluation value Ev obtained by replacing the depth value _{Du, v} smoothed by the smoothing means 80 with the depth value D ′.
Ev ′ (D ′, u, v) = E (D ′, u, v) + W {| _{Du + 1, v−} D ′ |
+ | _{Du , v + 1−} D ′ |} (24) Ev ′ = Σ _{x, y, Du, v → D ′} Ev (D _{x, y} , x, y) (25)

That is, the depth value determining unit 90 selects the selection difference value E (D ′, u, v) for each of the depth value D ′ and the target pixel P _C (pixel position (u, v)) according to the above equation (24). to the pixel of interest _{P C} to the adjacent pixel adjacent depth value _{D u + 1, v, D} u, and _{v + 1,} 'by weighted addition of the difference between the recalculated difference value Ev' depth value D (D ', u , V). Then, the depth value determining means 90 calculates the depth value D _{x, y} at the coordinates (u, v) of the selected difference value E (D _{x, y} , x, y) by the above-described equation (25). 'smoothed difference value Ev when replaced by _{(D x, y, x,} y) to that summing all pixels in the reference image F _C, recalculated evaluation value Ev' is calculated.

When the depth value D _{u, v is changed} to the depth value D ′, the value of the smoothing difference value Ev (D ′, u, v) at the coordinates (u, v) is changed, and the smoothing evaluation at adjacent pixels is performed. The difference in the depth value that is the smoothing term of the value Ev also changes. Therefore, the recalculated difference values Ev ′ (D ′, u, v) calculated by the equation (24) are not summed, but the smoothed difference values Ev (D _{x, y} , x, y) are summed (25). The formula is used to prevent the calculation formula from becoming too complex.

Then, the depth value determining unit 90 determines whether or not the recalculated evaluation value Ev ′ is less than the smoothing evaluation value Ev by the following equation (26).
Ev ′ <Ev (26)

When the recalculated evaluation value Ev ′ is less than the smoothed evaluation value Ev, the depth value determining means 90 replaces the value of the smoothed evaluation value Ev with the value of the recalculated evaluation value Ev ′ according to the following equation (27). The value of the depth value D _{u, v at} the pixel position (u, v) is replaced with the value of the depth value D ′. In the equation (27), “=” means that the value on the left side is replaced with the value on the right side.
Ev = Ev ′
D _{u, v} = D ′ (27)

  On the other hand, when the recalculated evaluation value Ev ′ is not less than the smoothed evaluation value Ev, the depth value determining unit 90 does not perform replacement according to the equation (27). It is assumed that the depth value determination unit 90 starts the operation when notified from the smoothing unit 80 that the smoothing of the selected difference value has been completed (smoothing completion).

Thus, depth value determining means 90, while changing the order of the depth values D 'from 0 to the maximum disparity, as smoothed evaluation value Ev in all the pixels of the reference image F _C decreases, each pixel position (u, By replacing the depth value D _{u, v} of v), the optimum depth value D _{u, v} can be obtained. Then, the depth value determination unit 90 outputs the depth image _{F Z} indicating the depth value _{D u, v} for each pixel position (u, v). Further, the depth determining means 90, the pixel position of the reference image F _C (u, v) for each depth value D _u, the _v, in the vicinity of the depth value output means 40 as a depth value D of each pixel of the reference image F _C Output.

  Note that the depth value D ′ is a value that assumes the depth value D of the subject T, as with the assumed depth value d, and takes a value from the minimum parallax amount to the maximum parallax amount. For example, when the depth value D is represented by 8-bit data, the depth value D ′ takes a value in the range of “0” to “255”.

As described above, the depth estimation apparatus 1 can estimate the depth value D of the subject T using the selection difference value as an index.
In addition, the depth estimation apparatus 1 performs the depth estimation after the 2nd time using the depth value D determined by the depth estimation of the 1st time. The depth estimation apparatus 1 may generate a depth image F _Z from the depth value D obtained after repeated depth estimation only repeat upper limit number N times. The repetition upper limit number N indicates the upper limit of the depth estimation number n, and is manually set to an arbitrary value.

[Operation of depth estimation device]
Hereinafter, the operation of the depth estimation apparatus 1 in FIG. 2 will be described with reference to FIGS. 8 to 16 (see FIG. 2 as appropriate). Here, the overall operation of the depth estimation apparatus 1 will be described with reference to FIG. 8, and the detailed description will be described with reference to FIGS. 9 to 16.

<Overall operation>
With reference to FIG. 8, the overall operation of the depth estimation apparatus 1 will be described.
The depth estimation apparatus 1 performs the initial setting with the depth estimation count n = 1 and the depth value D = 0 of all pixels. The depth estimation apparatus 1 is identical to the camera CM (reference camera C _C , adjacent cameras C _LL , C _L , C _R , and C _RR ) that are horizontally arranged at equal intervals on a straight line by the video input unit 10. subject T the captured camera image (reference image _{F C,} the neighboring image _{F LL, F L, F R} , F RR) to enter (step S1).

Depth estimation apparatus 1, the y-coordinate values and the x-coordinate value of the pixel of interest P _C is initialized to 0 (step S2, S3), it is initialized to 0 assumptions depth value d (step S4).

Depth estimation apparatus 1 by the corresponding pixel difference calculating unit 20, the pixel value of the pixel of interest P _C of the reference image F _C, the neighboring image F _L corresponding to the parallax assumptions depth value d of the pixel of interest P _C, F _R of the corresponding pixel _P L, _P corresponding pixel difference values of adjacent image which is a difference absolute value between the pixel value of _{R E CL,} calculates the _{E CR} (step S5; corresponding pixel difference calculating operation). The operation of step S5 will be described later.

  The depth estimation apparatus 1 determines either the minimum difference value Em or the average difference value Ea as a selected difference value by the difference value determination unit 60 (step S6; difference value determination operation). The operation of step S6 will be described later.

  The depth estimation apparatus 1 sequentially adds (increments) the assumed depth value d, which is a variable, the x-coordinate value of the camera image, and the y-coordinate value. After that, the depth estimation apparatus 1 repeats the operations in steps S5 and S6 until the assumed depth value d reaches the maximum parallax amount (steps S7 and S8), and the x coordinate value in steps S4 to S6 until the video width is reached. The operation is repeated (steps S9 and S10), and the operation of steps S3 to S6 is repeated for the y coordinate value until the image height is reached (steps S11 and S12). Note that the video height is manually set to an arbitrary value (for example, 1080 pixels), like the video width.

Depth estimation apparatus 1, the smoothing means 80, all pixel positions of the reference image F _C (x, y) for each, assuming the depth value d to correspondingly stored selected difference value E (d, x, y ) Is smoothed with the depth value of the adjacent pixel (step S13; smoothing operation). The operation of step S13 will be described later.

Depth estimation apparatus 1, the depth value determining means 90, the entire reference image F _C, the pixel position (x, y) the sum of the smoothed difference value for each is the minimum depth value D (x, y) a set of search to, for all pixel positions of the reference image F _C determines a depth value D (step S14; depth value determining operation). The operation of step S14 will be described later.

  The depth estimation apparatus 1 determines whether the depth estimation number n is less than the repetition upper limit number N (step S15). When the depth estimation number n is less than the repetition upper limit number N (Yes in step S15), the depth estimation device 1 proceeds to the process of step S16. In this case, the depth estimation apparatus 1 increments the depth estimation number n (step S16), and returns to the process of step S2.

On the other hand, if the depth estimation number n is not less than the repetition upper limit number N (No in step S15), and the depth estimation apparatus 1, the depth value determination unit 90, outputs the depth image F _Z indicating a depth value D for each pixel position (Step S17), and the operation ends. As described above, the depth estimation apparatus 1 can estimate the depth value D indicating the depth of the subject T from a plurality of camera images by the processes of steps S1 to S17.

<Corresponding pixel difference calculation operation>
With reference to FIG. 9, the corresponding pixel difference calculation operation (step S5 in FIG. 8) will be described in detail.

Corresponding pixel difference calculating means 20 is checked by the aforementioned formula (2), whether the corresponding pixel P _R is not deviated from neighboring image F _R (step S50). If the corresponding pixel _{P R} is not deviated from neighboring image _{F R} (Yes in step S50), the corresponding pixel difference calculating unit 20, the process proceeds to step S51. On the other hand, if the corresponding pixel _{P R} is out of the neighboring image _{F R} (No at step S50), the corresponding pixel difference calculating unit 20, the processing proceeds to step S54.

Corresponding pixel difference calculating means 20 is checked by the aforementioned equation (3), whether or not the corresponding pixel P _L does not deviate from neighboring image F _L (step S51). If the corresponding pixel _{P L} does not deviate from neighboring image _{F L} (Yes in step S51), the corresponding pixel difference calculating unit 20, the process proceeds to step S52. On the other hand, if the corresponding pixels _{P L} is out of the neighboring image _{F L} (No in step S51), the corresponding pixel difference calculating unit 20, the process proceeds to step S53.

Corresponding pixel difference calculating means 20, according to the above mentioned (1), the left and right neighboring image _F L, _{F R} on the assumption depth value d by the corresponding pixel has been moved to the right or left _P L, _{P R} of the pixel value _P L ( _{x + d, y), P} R (x-d, and y), the pixel value _P C (x of the pixel of interest _{P C,} y) and the corresponding pixel difference value _{E CL (d, x, y} ), E CR (d , X, y) is calculated (step S52).

Corresponding pixel difference calculating means 20, according to the above mentioned (4), the right neighboring image _{F R,} the corresponding pixel has moved to the left by assuming the depth value d in _{F RR P RR, P R} of the pixel value _P RR (x- 2d, y), P _R (x−d, y) and corresponding pixel difference values E _CL (d, x, y) and E _CR (d, x, y) are calculated (step S53).

The corresponding pixel difference calculation means 20 calculates the pixel value P _L (x + d, x) of the corresponding pixels P _L and P _LL moved to the right by the assumed depth value d in the left adjacent images F _L and F _LL according to the above-described equation (5). y), pixel difference values E _CL (d, x, y) and E _CR (d, x, y) corresponding to P _LL (x + 2d, y) are calculated (step S54).

That is, the corresponding pixel difference calculating means 20, the corresponding pixel _P L, _{P R} is the left and right neighboring image _F L, when within _{F R,} the left and right neighboring image _F L, with _{F R,} the corresponding pixel difference value E _CL and _ECR are calculated. When the corresponding pixel is deviated from one of the adjacent images F _L and F _R , the corresponding pixel difference calculation means 20 uses the two adjacent images (right adjacent image F _R on the right side) opposite to the adjacent image on the side from which the corresponding pixel is removed. , F _RR or left adjacent video images F _L , F _LL ), corresponding pixel difference values E _CL , E _CR are calculated. Thus, the depth estimation apparatus 1 has two corresponding pixel difference value E _CL, it is possible to determine the optimum choice difference value from E _CR, it is possible to estimate the correct depth value.

<Differential value determination operation>
With reference to FIG. 10, the difference value determining operation (step S6 in FIG. 8) will be described in detail (see FIG. 2 as appropriate).

  The difference value determination means 60 determines whether the depth estimation number n is the first time or the second time or later by the difference value selection means 60c (step S60). When the depth estimation number n is the first time (Yes in step S60), the difference value determination unit 60 proceeds to the process of step S61. On the other hand, when the depth estimation number n is the second time or later (No in step S60), the difference value determination unit 60 proceeds to the process of step S62.

  The difference value determining means 60 calculates the average difference value Ea (d, x, y) by the above-described equation (11) by the average difference value calculating means 60a. Then, the difference value determining means 60 selects the average difference value Ea as the selected difference value E by the difference value selecting means 60c as shown in the above-described equation (13) (step S61).

Difference determining means 60 acquires from the vicinity depth value output unit 40, neighboring pixels _{P SL,} the depth value _D L of _{P SR,} the _{D R} (step S62; near depth value output operation). The operation of step S62 will be described later.

  The difference value determination unit 60 selects either the minimum difference value Em or the average difference value Ea as the selection difference value E based on a preset condition by the difference value selection unit 60c (step S63; difference value). Selection action). The operation of step S63 will be described later.

<Near depth value output operation>
Referring to FIG. 11, the neighborhood depth value output manual operation (step S62 in FIG. 10) will be described in detail (see FIG. 2 as appropriate).

Near depth value output section 40 sets the number of neighboring pixels s in maximum disparity amount _{D M} (step S620). Then, near the depth value output means 40, according to the above mentioned (7), the left and right neighboring pixels _{P SL,} the _{P SR} depth value _D L, obtains the _{D R} (step S621).

Near depth value output means 40, according to the above mentioned (8), a difference absolute value between the left depth value _{D L} and the right depth value _{D R |} determines whether exceeds the threshold value th1 | D L -D _R (Step S622). If the difference absolute value | D _L −D _R | exceeds the threshold th1 (Yes in step S622), the neighborhood depth value output unit 40 ends the process. On the other hand, if the difference absolute value | D _L −D _R | does not exceed the threshold th1 (No in step S622), the neighborhood depth value output unit 40 proceeds to the process of step S623.

The neighborhood depth value output means 40 halves the number of neighboring pixels s by the above-described equation (9) (step S623). Then, near the depth value output unit 40, according to the above mentioned (7), the number of neighboring pixels s are neighboring pixels becomes half _{P SL,} the _{P SR} depth value _D L, obtains the _{D R} (step S624).

<Differential value selection operation>
With reference to FIG. 12, the difference value selection operation (step S63 in FIG. 10) will be described in detail (see FIG. 2 as appropriate). In the subsequent drawings, the pixel values of neighboring pixels may be shown as neighboring pixel values.

Difference value selection unit 60c acquires from the neighboring pixel value output unit 30, the pixel value of the pixel of interest _{P C,} neighboring pixel _{P SL,} and a pixel value of _{P SR} (step S630; neighboring pixel value output operation). The operation of step S630 will be described later.

The difference value selection means 60c calculates the absolute difference value E _LR of the pixel values of the left and right neighboring pixels P _SL and P _SR by the above-described equation (14) (step S631), and the absolute difference value | D _L −D _R It is determined whether or not | exceeds the threshold th1 and the difference absolute value E _LR exceeds the threshold th2 (step S632).

When the difference absolute value | D _L −D _R | exceeds the threshold th1 and the difference absolute value E _LR exceeds the threshold th2 (Yes in step S632), the difference value selection unit 60c proceeds to the process of step S633. On the other hand, if the difference absolute value | D _L −D _R | does not exceed the threshold th1, or if the difference absolute value E _LR does not exceed the threshold th2 (No in step S632), the difference value selection unit 60c performs step S639. Proceed to the process.

Difference value selection unit 60c from neighboring depth corresponding pixel value output unit 50, the pixel value _P LDL neighboring depth corresponding pixel in the pixel position corresponding to parallax of the depth value _{D L,} and _{P RDL,} parallax depth value _{D R} obtaining pixel values _P LDR neighboring depth corresponding _pixel, and a _{P RDR} at the corresponding pixel position (step S633; near depth corresponding pixel value output operation). The operation of step S633 will be described later.

Difference value selecting means 60c is the aforementioned (16), left and right neighboring image F _L, in F _R, is moved to the left and right by the number of pixels corresponding to the pixel of interest P _C to a depth value D _R Background A side calculating pixel value _P LDR neighboring depth corresponding _pixel, and _{P RDR,} differential absolute value _E LDR between the pixel value _{P C} of the pixel of _interest, the _{E RDR.} The moving difference value selecting unit 60c is the aforementioned (19), left and right neighboring image _F L, in _{F R,} the left and right by the number of pixels corresponding to the pixel of interest _{P C} to a depth value _{D L} background A side pixel value _P LDL neighboring depth corresponding pixels is _calculated and _{P RDL,} differential absolute value _E LDL pixel values between the target pixel _{P C,} the _{E RDL} (step S634).

Or the difference value selecting unit 60c is the aforementioned (15), by the depth value D _L is determined to or greater than the depth value D _R, there is a background A the depth value decreases in the left or right side Is determined (step S635). If the depth value _{D L} is greater than the depth value _{D R} (Yes in step S635), the difference value selecting unit 60c, the process proceeds to step S636. On the other hand, if the depth value _{D L} is less than the depth value _{D R} (No at step S635), the difference value selecting unit 60c, the process proceeds to step S637.

Difference value selection unit 60c is checked by the aforementioned (17), by the difference absolute value E _LDR to determine whether exceeds the difference absolute value E _RDR, the existence of occlusions in the left neighboring image F _L (Step S636). When the difference absolute value _{E LDR} exceeds the difference absolute value _{E RDR} (Yes in step S636), the difference value selecting unit 60c, the process proceeds to step S638. On the other hand, when the difference absolute value _{E LDR} does not exceed the difference absolute value _{E RDR} (No at step S636), the difference value selecting unit 60c, the process proceeds to step S639.

Difference value selection unit 60c is checked by the aforementioned equation (20), by the difference absolute value E _RDL to determine whether exceeds the difference absolute value E _LDL, the presence of occlusion in the right neighboring image F _R (Step S637). When the difference absolute value E _RDL exceeds the difference absolute value E _LDL (Yes in step S637), the difference value selection unit 60c proceeds to the process of step S638. On the other hand, if the difference absolute value E _RDL does not exceed the difference absolute value E _LDL (No in step S637), the difference value selection unit 60c proceeds to the process of step S639.

  The difference value selection means 60c calculates the minimum difference value Em (d, x, y) calculated by the minimum difference value selection means 60b using the above-described expressions (12) and (18) as the selection difference value E (d, x, y). y) is selected (step 638).

  The difference value selection means 60c calculates the average difference value Ea (d, x, y) calculated by the average difference value calculation means 60a according to the above-described expressions (11) and (13) as the selection difference value E (d, x, y). y) is selected (step 639).

  As described above, the difference value selection unit 60c corresponds to a plurality of assumed depth values d (“0” to the maximum parallax amount) for each pixel position (x, y) of the reference image by the processing in steps S630 to S639. The selected difference value E (d, x, y) can be selected.

<Neighboring pixel value output operation>
Referring to FIG. 13, the neighboring pixel value output manual operation (step S630 in FIG. 12) will be described in detail (see FIG. 2 as appropriate).

Neighboring pixel value output unit 30, according to the above mentioned (6), the pixel value _{P C} of the pixel of interest _P C (x, y), the pixel of interest _P C (x, y) spaced to the left only in the vicinity pixel number s from The pixel value P _SL of the neighboring pixel at the pixel position P _C (x−s, y) and the pixel position P _C (x + s, y) separated from the target pixel P _C (x, y) by the number of neighboring pixels s to the right. and acquires the pixel value _{P SR} neighboring pixels in) (step S6300).

<Neighbor Depth-Compatible Pixel Value Output Operation>
With reference to FIG. 14, the neighborhood depth corresponding pixel value output manual operation (step S <b> 633 in FIG. 12) will be described in detail.

The neighborhood depth corresponding pixel value output means 50 calculates the neighborhood depth correspondence pixel P _L (x + D) at the pixel position corresponding to the parallax of the depth value D _L from the left and right adjacent images F _L and F _{R according} to the above-described equation (10). _{L, y), P R (} x-D L, y) of the pixel values _{P LDL,} and calculates the _{P RDL.} Then, near the depth corresponding pixel value output unit 50, according to the above mentioned (10), left and right neighboring image F _L, from F _R, the depth value D near depth corresponding pixel is in the pixel position corresponding to parallax of _R P _L Pixel values P _LDR and P _RDR of (x + D _R , y) and P _R (x−D _R , y) are calculated (step S6330).

<Smoothing operation>
The smoothing operation (step S13 in FIG. 8) will be described in detail with reference to FIG.

The smoothing means 80 initializes (x = 0, y = 0) the x coordinate value and the y coordinate value of the camera image (reference image F _C ) as variables (steps S130 and S131).

The smoothing unit 80 selects the difference value E (d, x, y) of the target pixel P _C for each target pixel P _C (pixel position (x, y)) of the reference image F _C according to the above-described equation (21). ) Is calculated as the assumed depth value d. Then, the smoothing unit 80 sets the assumed depth value d as the depth value D _{x, y of the} pixel (step S132).

  The smoothing means 80 sequentially adds (increments) the x-coordinate value and the y-coordinate value of the camera image that are variables. After that, the smoothing means 80 repeats the operation of step S132 until the video width becomes the x coordinate value (steps S133 and S134), and repeats the operations of steps S131 and S132 until the video height is reached for the y coordinate value ( Steps S135 and S136).

Smoothing means 80, according to the above mentioned (21), a pixel position adjacent to the pixel of interest _{P C (x + 1, y} ), (x, y + 1) selection difference values of adjacent pixels becomes the minimum assumed depth value _{D x + 1 , Y} , D _{x, y + 1} are calculated. Then, the smoothing unit 80, by (22), the pixel of interest _{P C} of the selected difference value _{E (D x, y, x} , y) , the assumption depth value _{D x + 1, y, D} x, y + 1 and the depth value D _x, by weighted addition of the difference between _y, and calculates selected difference value _{E (D x, y, x} , y) smoothing the smoothed difference value _{Ev (D x, y, x} , y) and (Step S137).

Smoothing means 80, according to the above mentioned (23), the summing pixel position (x, y) for each of the smoothed difference value _{Ev (D x, y, x} , y) of all pixels of the reference image _{F C} Thus, the smoothing evaluation value Ev is calculated (step S138).

<Depth value determination operation>
With reference to FIG. 16, the depth value determining operation (step S14 in FIG. 8) will be described in detail.

Depth value determining means 90, the depth value of all the pixels (the second assumed depth value) D 'initialized (S140), the v coordinate value and u coordinate value of the pixel of interest _{P C} is initialized to 0 (step S141 , S142).

Depth value determining means 90, according to the above mentioned (24) and (25) below, while changing the second assumption depth value D 'to the maximum amount of parallax from the minimum amount of parallax pixel position of the reference image F _C (u, For each v), the depth value D _u smoothed by the smoothing means 80 for the selection difference value E (D ′, u, v) stored in the storage means 70 in association with the pixel position (u, v). _{, V} is replaced with a depth value D ′, a recalculated evaluation value Ev ′ is calculated (step S143).

  The depth value determining means 90 determines whether or not the recalculated evaluation value Ev ′ is less than the smoothing evaluation value Ev by the above-described equation (26) (step S144). When the recalculated evaluation value Ev ′ is less than the smoothed evaluation value Ev (Yes in step S144), the depth value determining unit 90 proceeds to the process of step S145. On the other hand, when the recalculated evaluation value Ev ′ is not less than the smoothed evaluation value Ev (No in step S144), the depth value determining unit 90 proceeds to the process of step S146.

The depth value determining means 90 replaces the value of the smoothed evaluation value Ev with the value of the recalculated evaluation value Ev ′ according to the above-described equation (27), and the depth value D _{u, v} of the pixel position (u, v). The value is replaced with the value of the depth value D ′ (step S145).

  The depth value determination means 90 sequentially adds (increments) the depth value D ′, u coordinate value, and v coordinate value, which are variables. Thereafter, the depth value determining means 90 repeats the operations in steps S143 to S145 until the video width is obtained for the u coordinate value (steps S146 and S147), and the operations in steps S142 to S145 are performed until the video height is obtained for the v coordinate value. Are repeated (steps S148 and S149), and the operation of steps S141 to S145 is repeated until the depth value D ′ reaches the maximum amount of parallax (steps S150 and S151).

  As described above, the depth estimation device 1 performs the pixel value of the target pixel, the pixel value of the neighboring pixel, the depth value of the neighboring pixel, and the neighboring depth corresponding to the depth value of the neighboring pixel regardless of the presence or absence of occlusion. Based on the pixel value of the corresponding pixel, the optimum one of the average difference value and the minimum difference value is selected. Thereby, the depth estimation apparatus 1 can estimate the depth value D from the more correct selection difference value by selecting the average difference value when false matching occurs in the adjacent video, and if there is occlusion, then Can be excluded from the estimation target. That is, the depth estimation device 1 can estimate an accurate depth value D.

In the first embodiment, as illustrated in FIG. 1, five videos (F _LL , F _L , F _C , F _R , F _RR ) have been described as being input to the depth estimation device 1. Is not limited to this. Hereinafter, a depth estimation system SB that estimates the depth value D of the subject T from three images (F _L , F _C , F _RR ) will be described as a second embodiment.

(Second Embodiment)
[Overall configuration of depth estimation system]
With reference to FIG. 17, the overall configuration of the depth estimation system SB including the depth estimation device 1B according to the second embodiment of the present invention will be described with respect to differences from the first embodiment.

As shown in FIG. 17, the depth estimation system SB includes a camera CM (C _L , C _C , C _R ) and a depth estimation device 1B. That is, in the depth estimation system SB, camera _C L, _{C R} are arranged so as to sandwich the camera _{C C} from left and right. Then, the images (F _L , F _C , F _RR ) captured by the camera CM (C _L , C _C , C _R ) are input to the depth estimation device 1B.

[Configuration of depth estimation device]
With reference to FIG. 18, the difference of the configuration of the depth estimation device 1B from the first embodiment will be described (see FIG. 17 as appropriate).

The depth estimation apparatus 1B is a depth indicating the depth of the subject T from the camera video (F _L , F _C , F _R ) obtained by photographing the subject T with the cameras CM (C _L , C _C , C _R ) arranged in the horizontal direction. Estimate the value D. For this reason, the depth estimation apparatus 1B includes a difference between the video input unit 10B, the corresponding pixel difference calculation unit 20B, the neighboring pixel value output unit 30, the neighboring depth value output unit 40, and the neighboring depth corresponding pixel value output unit 50. Value determining means 60, storage means 70, smoothing means 80, and depth value determining means 90 are provided.

The video input unit 10B inputs a plurality of camera videos taken by the camera CM. Here, the image input unit 10B is input and the reference image _{F C} taken by the base camera _{C C,} adjacent camera _C L, neighboring image _F L taken by _{C R,} and _{F R.}
Each video (F _L , F _C , F _R ) input by this video input means 10B is stored in a memory (not shown), and corresponding pixel difference calculation means 20B, neighborhood pixel value output means 30, and neighborhood The depth corresponding pixel value output means 50 shall be referred to.

Corresponding pixel difference calculation unit 20B, depending aforementioned (2), with the corresponding pixel P _R to determine whether or not deviate from neighboring image F _R, according to the above mentioned (3), the corresponding pixel P _L is It determines whether or not deviate from neighboring image F _L. Then, the corresponding pixel difference calculation unit 20B, the neighboring image _F L, the corresponding pixel _P L from _{F R,} if not disconnected _{P R,} i.e., if it meets both the (2) and (3), and the ( Corresponding pixel difference values E _CL and E _CR are calculated by the equation 1).

If the right neighboring image _{F R} corresponding pixel _{P R} deviates, i.e., (2) is not satisfied equation corresponding pixel difference calculation unit 20B, the following expression (28) by the maximum amount of parallax corresponding pixel difference _{E CR} (For example, “255”).
E _CL (d, x, y) = | P _L (x + d, y) −P _C (x, y) |
E _CR (d, x, y) = 255 (28)

If the neighboring image _{F L} corresponding pixel _{P L} is disengaged, i.e., (3) is not satisfied equation corresponding pixel difference calculation unit 20B, the following expression (29) by the maximum amount of parallax corresponding pixel difference value _{E CL} (For example, “255”).
E _CL (d, x, y) = 255
E _CR (d, x, y) = | P _R (x−d, y) −P _C (x, y) | (29)

[Operation of depth estimation device]
With reference to FIG. 19, a corresponding pixel difference calculation operation (step S5 in FIG. 8) will be described as an operation of the depth estimation device 1B (see FIG. 18 as appropriate). Note that the processing in steps S50 to S52 in FIG. 19 is the same as each processing in FIG.

Corresponding pixel difference calculation unit 20B, when the corresponding pixel _{P L} is out of the neighboring image _{F L} (No in step S51), the aforementioned (29), calculates a corresponding pixel difference value _{E CL} as the maximum amount of parallax (Step S53 ′).

Corresponding pixel difference calculation unit 20B, the corresponding pixel _{P R} is the previously described (28) (No in step S50) if deviates from neighboring image _{F R,} and calculates the corresponding pixel difference value _{E CR} as the maximum amount of parallax ( Step S54 ').

As described above, the depth estimation apparatus 1B according to the second embodiment of the present invention can estimate the depth value D of the subject T from the three videos (F _L , F _C , F _R ). Therefore, the depth estimation apparatus 1B can reduce the erroneous estimation of the depth value as in the first embodiment.

  Although the configuration and operation of the depth estimation device 1B according to the second embodiment of the present invention have been described, the present invention is not limited to this embodiment. Hereinafter, various modifications of the present invention will be described.

(Modification 1: Modification of camera arrangement interval)
In the second embodiment, the installation intervals of the cameras that photograph the subject are set to be equal intervals, but different camera intervals may be used.
In this case, the corresponding pixel difference calculating unit 20B, each neighboring image F _L, the F _R, for each camera distance from the base camera C _C, from the pixels of the reference image F _C, the positions of the pixels corresponding to the parallax, respectively determined, it may be operational and the pixel value of the pixel, a difference absolute value between the pixel value of the pixel of the reference image F _C. For example, in this embodiment, the unit of the depth value D and the unit of parallax between the pixels are the same, and when the depth value D increases by “1”, the parallax increases by one pixel. the camera _C C shown in FIG. 17, the distance between _{C R,} camera _C C, when assumed to be twice the distance between _{C L,} the corresponding pixel difference calculation unit 20B instead of the (1) The difference absolute value may be calculated by the following equation (30).
E _CL (d, x, y) = | P _L (x + d, y) −P _C (x, y) |
E _CR (d, x, y) = | P _R (x−2d, y) −P _C (x, y) | (30) Thus, the degree of freedom of camera arrangement of the depth estimation system (FIG. 17). Can be increased.

(Modification 2: Modification of the corresponding pixel difference calculation means)
Further, in the second embodiment, the corresponding pixel difference calculation unit 20 </ b> B uses the pixel value of each pixel of the reference image F _C and each pixel of the adjacent images F _L and F _R corresponding to the parallax of the assumed depth value d of the pixel. The difference absolute value with respect to the pixel value is calculated using the weighted average value of the difference absolute value of the pixel values in the block centered on the corresponding pixel as well as the corresponding pixel. It is good as well. For example, the corresponding pixel difference calculation unit 20B calculates the corresponding pixel difference value E _CL (d, x, y) of the equation (1) according to the following equation (31).
E _CL (d, x, y) = | P _L (x + d, y) −P _C (x, y) | × 1/2
+ | P _L (x + d + 1, y) −P _C (x + 1, y) | × 1/8
+ | P _L (x + d-1, y) -P _C (x-1, y) | × 1/8
+ | P _L (x + d, y + 1) −P _C (x, y + 1) | × 1/8
+ | P _L (x + d, y−1) −P _C (x, y−1) | × 1/8
... (31) Thereby, the difference | error of the absolute difference value generate | occur | produced by noise etc. can be reduced.

(Modification 3: Modification of difference value selection means)
In each embodiment, the average difference value calculation unit 60a calculates the average difference value Ea and the minimum difference value selection unit 60b calculates the minimum difference value Em. However, the present invention is not limited to this. . For example, in the present invention, the difference value selection unit 60c may calculate the average difference value Ea and the minimum difference value Em.

(Modification 4: Modification of difference value selection means)
In each embodiment, the difference value selection unit 60c has been described as using the above-described equations (17) and (20), but the present invention is not limited to this. For example, in the present invention, the difference value selection means 60c may use a magnification k set to an arbitrary value in advance.

Specifically, the difference value selection means 60c determines whether or not the difference absolute value _ERDR exceeds the difference absolute value _ERDR after multiplying the right side of the above (17) by the magnification k ((Expression 32) )reference).
E _LDR > k × E _RDR (32) formula

The difference value selection means 60c determines whether or not the difference absolute value E _RDL exceeds the difference absolute value E _LDL after multiplying the left side of the above expression (20) by the magnification k (see (33)). ).
E _LDL × k <E _RDL (33) Thereby, the difference value selection means 60c can more accurately determine the presence or absence of occlusion.

(Modification 5: Modification of Difference Value Selection Operation)
In each embodiment, in the difference value selection operation (see FIG. 12), the difference absolute values E _LDR , E _RDR , E _LDL , and E _RDL are calculated (step S634), and then the processing after step S635 is executed. However, the present invention is not limited to this. Here, a case is considered in which the difference value selection unit 60c executes the processing up to step S633 in the difference value selection operation.

In this case, the difference value selecting unit 60c, similar to the processing in step S635, according to the above mentioned (15), determines whether a depth value _{D L} is greater than or equal to the depth value _{D R.} If (15) is established, the difference value selecting unit 60c is the aforementioned (16), near the depth corresponding pixel _{P LDR,} a pixel value of _{P RDR,} differential absolute value E of the pixel value of the pixel of interest _{P C LDR} and _ERDR are calculated. Then, the difference value selection unit 60c executes the processing after step S636.

On the other hand, if (15) is not satisfied, the difference value selecting unit 60c is the aforementioned (19), near the depth corresponding pixel _{P LDL,} and the pixel values of _{P RDL,} the difference between the pixel value of the pixel of interest _{P C} Absolute The values E _LDL and E _RDL are calculated.

Thus, the difference value selecting unit 60c is the depth value _D L, after performing a size determination of _{D R,} according to the determination result, the difference absolute value _{E LDR, E RDR} or absolute difference _{E LDL, E RDL} May be calculated.

  In each embodiment, the depth estimation apparatus according to the present invention has been described as an independent apparatus, but the present invention is not limited to this. For example, in the present invention, a general computer hardware resource can be realized by a depth estimation program that performs a cooperative operation as each unit of the depth estimation device. The depth estimation program can also be distributed via a communication line, or can be recorded on a recording medium such as a CD-ROM for distribution.

1, 1B Depth estimation device 10, 10B Video input means 20, 20B Corresponding pixel difference calculation means 30 Neighborhood pixel value output means 40 Neighborhood depth value output means 50 Neighborhood depth corresponding pixel value output means 60 Difference value determination means 60a Average difference value calculation Means 60b Minimum difference value selection means 60c Difference value selection means 70 Storage means 80 Smoothing means 90 Depth value determination means

Claims (9)

A depth estimation device that estimates a depth value indicating the depth of the subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position,
Video input means for inputting a reference image obtained by photographing the subject with a predetermined reference camera and a plurality of adjacent images obtained by photographing the subject with an adjacent camera other than the reference camera in any one of the plurality of cameras arranged in the plurality. When,
For each of the assumed depth value assuming the depth value and the target pixel of the reference image, the pixel value of the target pixel and the plurality of adjacent images at pixel positions corresponding to the parallax of the assumed depth value of the target pixel. A corresponding pixel difference calculating means for calculating a corresponding pixel difference value which is an absolute difference between the corresponding pixel value and the pixel value;
In the reference video input to the video input means, the pixel value of the target pixel of the reference video and the pixel value of the neighboring pixel at the pixel position separated from the target pixel to the left and right by the predetermined number of neighboring pixels are output. Neighboring pixel value output means for
For each of the assumed depth value and the target pixel, a neighboring depth that outputs a pixel value of a neighboring depth-corresponding pixel at a pixel position corresponding to a parallax of the neighboring pixel depth value from the adjacent images on the left and right with respect to the reference image. Corresponding pixel value output means;
For each of the assumed depth value and the target pixel, an average difference value that is an average value of the corresponding pixel difference values and a minimum difference value that is a minimum value of the corresponding pixel difference values are obtained, and the pixel value of the target pixel and the Based on a pixel value of a neighboring pixel, a depth value of the neighboring pixel, and a pixel value of the neighboring depth corresponding pixel, either the minimum difference value or the average difference value is determined as a selection difference value according to a predetermined condition. Difference value determining means to perform,
Storage means for storing the selection difference value determined by the difference value determination means in association with the assumed depth value and the pixel position of the reference image;
For each target pixel, the selection difference value of the target pixel includes an assumed depth value at which the selection difference value of a pixel adjacent to the target pixel is minimum and an assumed depth value at which the selection difference value of the target pixel is minimum. Smoothing means for calculating a smoothed evaluation value that is a sum of the smoothed difference values, while calculating a smoothed difference value obtained by smoothing the selected difference value by weighted addition of the differences;
A recalculated evaluation value corresponding to the smoothed evaluation value by replacing the assumed depth value corresponding to the selected difference value stored in the storage means with a second assumed depth value assuming a depth value different from the assumed depth value. When the recalculated evaluation value is less than the smoothed evaluation value calculated by the smoothing means, the second assumed depth value is determined as the depth value of each pixel of the reference image, and the recalculated evaluation If the value is not less than the smoothing evaluation value, a depth value determination unit that determines an assumed depth value corresponding to the selection difference value stored in the storage unit as a depth value of each pixel of the reference image;
A depth value of each pixel of the reference image is input from the depth value determining means, and a depth value output means for outputting the depth value of the neighboring pixels to the neighboring depth corresponding pixel value output means and the difference value determining means; ,
A depth estimation apparatus comprising:   The difference value determining means is configured such that the absolute difference value of the depth value between the neighboring pixels is equal to or smaller than a predetermined first threshold value, or the second absolute difference value of the pixel value between the neighboring pixels is predetermined. The depth estimation apparatus according to claim 1, wherein the average difference value is determined as the selection difference value when the threshold value is equal to or less than a threshold value. The difference value determining means includes a second threshold value in which a difference absolute value of a depth value between the neighboring pixels exceeds a predetermined first threshold value, and a difference absolute value of a pixel value between the neighboring pixels is set in advance. When calculating the absolute value of the difference between the pixel value of the neighboring depth corresponding pixel and the pixel value of the target pixel, from each of the left and right adjacent video with respect to the reference video,
The difference absolute value between the pixel value of the neighboring depth corresponding pixel on the side having the smaller depth value of the left and right neighboring pixels and the pixel value of the target pixel is the neighboring depth corresponding pixel on the opposite side to the smaller one The depth estimation apparatus according to claim 1, wherein the minimum difference value is determined as the selection difference value when the difference is smaller than the absolute difference between the pixel value of the pixel of interest and the pixel value of the target pixel. The difference value determining means includes a second threshold value in which a difference absolute value of a depth value between the neighboring pixels exceeds a predetermined first threshold value, and a difference absolute value of a pixel value between the neighboring pixels is set in advance. When calculating the absolute value of the difference between the pixel value of the neighboring depth corresponding pixel and the pixel value of the target pixel, from each of the left and right adjacent video with respect to the reference video,
The difference absolute value between the pixel value of the neighboring depth corresponding pixel on the side having the smaller depth value of the left and right neighboring pixels and the pixel value of the target pixel is the neighboring depth corresponding pixel on the opposite side to the smaller one The depth estimation device according to claim 1, wherein the average difference value is determined as the selection difference value when the difference value is not smaller than the absolute difference value between the pixel value of the target pixel and the pixel value of the target pixel. The difference value determining means determines the average difference value as the selected difference value when the depth count is estimated for the first time,
The depth estimation apparatus according to claim 1, wherein when the depth value is estimated for the second time or later, the selection difference value is determined according to the predetermined condition.   The corresponding pixel difference calculation means, when the corresponding pixel is deviated from the adjacent video located on the left and right of the reference video, with respect to the reference video, the adjacent video on the opposite side to the adjacent image on the side where the corresponding pixel is removed, and The difference absolute value of the pixel value of the corresponding pixel is calculated as the corresponding pixel difference value in an adjacent video further adjacent to the adjacent video on the opposite side. The depth estimation apparatus according to the item.   The neighboring depth value output means divides the number of neighboring pixels by half when the difference absolute value of the depth value between neighboring pixels is equal to or smaller than a predetermined threshold, and is a pixel that is separated from the target pixel by the number of neighboring pixels to the left and right The depth estimation apparatus according to any one of claims 1 to 6, wherein a depth value of a pixel in the vicinity of the position is output. A depth estimation method for estimating a depth value indicating the depth of a subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged in a predetermined position,
Video input means, in any one of the plurality of cameras arranged, a reference video obtained by photographing the subject with a predetermined reference camera and a plurality of adjacent videos obtained by photographing the subject with an adjacent camera other than the reference camera. Video input step to input,
Corresponding pixel difference calculation means, for each hypothetical depth value assuming the depth value and the target pixel of the reference image, at the pixel position corresponding to the pixel value of the target pixel and the parallax of the assumed depth value of the target pixel A corresponding pixel difference calculation step of calculating a corresponding pixel difference value that is a difference absolute value with a pixel value of a corresponding pixel of the plurality of adjacent videos
The difference value determining means obtains an average difference value that is an average value of the corresponding pixel difference values and a minimum difference value that is a minimum value of the corresponding pixel difference values for each of the assumed depth value and the target pixel, and the pixel value of the pixel, the pixel values of the neighboring pixels neighboring pixel number only to the pixel position away to the left and right from the pixel of interest a predetermined, and depth values of the neighboring pixels corresponding to the disparity of depth values of the neighboring pixels Based on the pixel value of the neighboring depth corresponding pixel at the pixel position to be determined, either the minimum difference value or the average difference value is determined as a selection difference value by a predetermined condition, the selection difference value A difference value determining step for writing to the storage means in association with the assumed depth value and the pixel position of the reference image;
For each target pixel, the smoothing unit minimizes the selection depth value of the target pixel, the assumed depth value at which the selection difference value of the pixel adjacent to the target pixel is minimum, and the selection difference value of the target pixel. A smoothing step of calculating a smoothed difference value obtained by smoothing the selected difference value by calculating a smoothed evaluation value that is a sum of the smoothed difference values by weighting and adding a difference with the assumed depth value. ,
The depth value determining means replaces the assumed depth value corresponding to the selected difference value stored in the storage means with a second assumed depth value in which a depth value different from the assumed depth value is assumed, thereby obtaining the smoothed evaluation value. The corresponding recalculated evaluation value is calculated, and when the recalculated evaluation value is less than the smoothed evaluation value calculated by the smoothing means, the second assumed depth value is determined as the depth value of each pixel of the reference video If the recalculated evaluation value is not less than the smoothed evaluation value, a depth value determination step of determining an assumed depth value corresponding to the selection difference value stored in the storage means as a depth value of each pixel of the reference image. And comprising
The difference value determining step includes:
A neighborhood depth value output means for outputting a depth value of the neighborhood pixel;
Neighboring pixel value output means, in the reference image, the pixel value of the target pixel of the reference image, and the neighboring pixel value output step of outputting the pixel values of the neighboring pixels,
Near depth corresponding pixel value output means, wherein each hypothetical depth value and the pixel of interest, from the left and right neighboring image with respect to the reference image, before KiKon near depth near depth corresponding pixel value output pixel value of the corresponding pixel An output step;
A depth estimation method comprising: In order to estimate a depth value indicating the depth of the subject from a plurality of camera images obtained by photographing the same subject with a plurality of cameras arranged at predetermined positions, a computer is provided.
Video input means for inputting a reference image obtained by photographing the subject with a predetermined reference camera and a plurality of adjacent images obtained by photographing the subject with an adjacent camera other than the reference camera in any one of the plurality of cameras arranged in the plurality. ,
For each of the assumed depth value assuming the depth value and the target pixel of the reference image, the pixel value of the target pixel and the plurality of adjacent images at pixel positions corresponding to the parallax of the assumed depth value of the target pixel. Corresponding pixel difference calculation means for calculating a corresponding pixel difference value that is an absolute difference value with respect to the pixel value of the corresponding pixel;
In the reference video input to the video input means, the pixel value of the target pixel of the reference video and the pixel value of the neighboring pixel at the pixel position separated from the target pixel to the left and right by the predetermined number of neighboring pixels are output. Neighboring pixel value output means for
For each of the assumed depth value and the target pixel, a neighboring depth that outputs a pixel value of a neighboring depth-corresponding pixel at a pixel position corresponding to a parallax of the neighboring pixel depth value from the adjacent images on the left and right with respect to the reference image. Corresponding pixel value output means,
For each of the assumed depth value and the target pixel, an average difference value that is an average value of the corresponding pixel difference values and a minimum difference value that is a minimum value of the corresponding pixel difference values are obtained, and the pixel value of the target pixel and the Based on a pixel value of a neighboring pixel, a depth value of the neighboring pixel, and a pixel value of the neighboring depth corresponding pixel, either the minimum difference value or the average difference value is determined as a selection difference value according to a predetermined condition. Then, a difference value determination unit that writes the selected difference value in the storage unit in association with the assumed depth value and the pixel position of the reference video,
For each target pixel, the selection difference value of the target pixel includes an assumed depth value at which the selection difference value of a pixel adjacent to the target pixel is minimum and an assumed depth value at which the selection difference value of the target pixel is minimum. Smoothing means for calculating a smoothed evaluation value which is a sum of the smoothed difference values, while calculating a smoothed difference value obtained by smoothing the selected difference value by weighted addition of the differences;
A recalculated evaluation value corresponding to the smoothed evaluation value by replacing the assumed depth value corresponding to the selected difference value stored in the storage means with a second assumed depth value assuming a depth value different from the assumed depth value. When the recalculated evaluation value is less than the smoothed evaluation value calculated by the smoothing means, the second assumed depth value is determined as the depth value of each pixel of the reference image, and the recalculated evaluation If the value is not less than the smoothing evaluation value, a depth value determination unit that determines an assumed depth value corresponding to the selection difference value stored in the storage unit as a depth value of each pixel of the reference image;
A depth value of each pixel of the reference image is input from the depth value determination means, and a depth value output means for outputting the depth value of the neighboring pixels to the neighboring depth corresponding pixel value output means and the difference value determination means,
Depth estimation program to function as

Images