JP4958302B2 - Multi-viewpoint image depth value extraction apparatus, method and program thereof - Google Patents

Description

  The present invention relates to a technique for extracting a depth value for a multi-view image (multi-view video) of a subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras.

  In recent years, various techniques for generating a stereoscopic video and a free viewpoint video using a plurality of captured images (multi-view images) obtained by capturing the same subject with a plurality of cameras have been disclosed (for example, see Patent Document 1). In these technologies for generating a stereoscopic video and a free viewpoint video, the depth value to the subject is basically obtained by using the parallax amount of the captured images taken by a plurality of cameras.

In addition, in order to obtain the parallax amount of the photographed image, a block similar to the photographed image in units of a predetermined size is searched between the photographed images (block matching), and the distance between the blocks is set as the parallax amount. Is common (see, for example, Patent Document 2). According to the method for obtaining the parallax by this block matching, as shown in FIG. 7, between the image M ₁ photographed by two horizontally arranged cameras (C ₁ , C ₂ ) and the image M ₂ , A corresponding region is searched by block matching, and the difference is calculated as a parallax amount (parallax vector) B.

Here, the parallax amount B is the block position T ₁ showing the photographed part of the subject by the camera C _1, corresponding to the block position T ₂ showing the photographed part of the same subject by the camera C _2, camera C ₁ Is a relative shift amount with respect to a block position T ₁ indicating the portion of the subject at (this position is T _{X in the} camera C ₂ ). That is, as shown in FIG. 7, the distance of the distance between the camera _{C 1} and the camera _{C 2} L, the camera _C 1, the focal length of _{C 2} f, the camera _C 1, _{C 2} to the subject S (depth value) Is D, and relative to the distance L between the cameras C ₁ and C ₂ , the subject S exists at the position (L−X): X. From the principle of general triangulation, the amount of parallax B is obtained by the following equation (1).
B = f (L−X) / D + fX / D = fL / D (1)

The parallax amount B is the parallax amount of the block T _{1 as well as} the parallax amount of the block T ₂ . From this equation (1), the distance (depth value) D from the cameras C ₁ and C ₂ to the subject S can be obtained by the following equation (2).
D = fL / B (2) As described above, conventionally, in order to obtain the depth value of a subject, a method of obtaining a parallax amount for each predetermined block and converting it to a depth value in a plurality of captured images is generally used. It is.
JP 2003-284097 A Japanese Patent Laid-Open No. 10-191393

  In the above-described conventional technique, in the method of obtaining the parallax amount by block matching between captured images captured by a plurality of cameras, if the subject looks different for each camera, a portion similar to the subject exists in the captured image. In such a case, there is a problem that a wrong block is searched and a wrong amount of parallax is extracted.

  Furthermore, in the prior art, since a plurality of captured images are captured by different cameras, corresponding points are not completely matched even by block matching, and there are many parallax amounts and depth values obtained thereby. There is a problem that errors are included.

  The present invention has been made in order to solve the above-described problems.Even if the corresponding points do not coincide with each other, the error can be reduced, and a plurality of captured images obtained by capturing the subject can be used. It is an object of the present invention to provide a multi-viewpoint image depth value extraction device, a method thereof, and a program thereof that can accurately extract depth values for a multi-viewpoint image.

  The present invention was devised to achieve the above object. First, the multi-viewpoint image depth value extraction device according to claim 1 is a plurality of captured images obtained by capturing the same subject with a plurality of cameras arranged. A multi-viewpoint image depth value extracting device for extracting a depth value for a multi-viewpoint image of the subject from the block, a block dividing unit, a block searching unit, a difference calculating unit, an average parallax amount calculating unit, and a depth value And a conversion means.

  In such a configuration, the multi-viewpoint image depth value extracting device divides a plurality of captured images into blocks having a predetermined size (for example, 16 × 16 pixels) by the block dividing unit. The size of the block serves as a reference for extracting the depth value, and can be arbitrarily determined depending on the resolution at which the depth value is extracted.

  Then, the multi-viewpoint image depth value extraction device uses a block search unit for each target block from which a depth value is to be extracted, for a plurality of references captured by a camera adjacent to the camera that captured the captured image to which the target block belongs. In the photographed image, a similar image region having the highest image feature similarity is searched. As a result, a region corresponding to the target block is searched for in the reference captured image.

Then, the multi-viewpoint image depth value extraction device calculates the difference between the position of the similar image region and the position of the target block by the difference calculation means. Further, in the multi-viewpoint image depth value extraction device, the average parallax amount calculating unit calculates the average value of the differences for each reference photographed image calculated by the difference calculating unit, and sets it as the parallax amount in the target block. Thus, the parallax amount of the target block is averaged for each reference captured image, and the error is dispersed.
At this time, the average parallax amount calculating means excludes the difference corresponding to the similar image area whose similarity with the target block is lower than a predetermined value from the value for calculating the average value, and calculates the average parallax amount. Ask. As described above, when the similarity between the similar image region and the target block is low, the amount of parallax with less error can be obtained by excluding the average value from the value for calculating the average value.

  Then, the multi-viewpoint image depth value extracting device converts the parallax amount calculated by the average parallax amount calculating unit to the target block based on the predetermined focal length of the camera and the distance between the cameras by the depth value converting unit. Convert to the depth value of the corresponding subject. The conversion from the parallax amount to the depth value can be performed by the calculation of the above-described equation (2).

The multi-viewpoint image depth value extraction device is a multi-viewpoint image depth value extraction device that extracts depth values for the multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged. A block dividing unit that divides the plurality of captured images into blocks of a predetermined size, and a target block for which the depth value of the captured image divided by the block dividing unit is to be extracted. In a plurality of reference captured images captured by a camera adjacent to the camera that captured the captured image to which the block belongs, a block search unit that searches for a similar image region having the highest image feature similarity, and the block search unit Difference calculating means for calculating a difference between the position of the similar image area and the position of the target block, and the reference photographed image portion calculated by the difference calculating means. An average parallax amount calculating unit that calculates an average value of the differences between the target blocks and the parallax amount calculated by the average parallax amount calculating unit for each captured image. A depth value converting means for converting into a depth value of the subject corresponding to the target block based on the focal length of the camera and the distance between the cameras, and a block spanning the similar image region in a plurality of the reference photographed images. Peripheral block specifying means for specifying as a peripheral block, and a similar image area depth for calculating an average value of the depth values corresponding to the peripheral blocks specified by the peripheral block specifying means and setting the depth value corresponding to the similar image area An average value of the depth value calculated by the value calculating means and the depth value calculating means of the similar image area and the depth value of the target block is calculated, and the target block is calculated. An average depth value calculating means for the depth value of the click, may be configured to include a.

At this time, the multi-viewpoint image depth value extraction device, kind similar image region depth value calculating means, when calculating the average value of the depth values corresponding to the peripheral block, determined in advance around the block with respect to the mean value When a depth value having a larger difference than the difference corresponds, the average value may be calculated again by excluding the depth value (claim 2).

  In such a configuration, the similar image region depth value calculation means calculates an average value by excluding peripheral blocks having greatly different depth values from the peripheral blocks.

Further, when the multi-viewpoint image depth value extraction device, the average depth value calculating means, for calculating an average value of the depth value of the calculated depth value and the target block in the similar image region depth value calculating means, similar image It is good also as a structure which excludes the depth value with a difference larger than a predetermined difference with respect to the said average value from the depth value calculated by the area | region depth value calculation means, and calculates an average value again (Claim 3).

  In such a configuration, the multi-viewpoint image depth value extraction device calculates an average value by excluding similar image regions having greatly different depth values from similar image regions.

The multi-viewpoint image depth value extracting method according to claim 4 is a multi-viewpoint image extracting a depth value for a multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged. The depth value extraction method is a procedure including a block division step, a block search step, a difference calculation step, an average parallax amount calculation step, and a depth value conversion step.

  In this procedure, the multi-viewpoint image depth value extraction method divides a plurality of captured images into blocks of a predetermined size in a block division step. In the multi-viewpoint image depth value extraction method, in the block search step, for each target block from which the depth value is to be extracted, a plurality of references captured by a camera adjacent to the camera that captured the captured image to which the target block belongs. In the photographed image, a similar image region having the highest image feature similarity is searched.

In the multi-viewpoint image depth value extraction method, the difference calculation step calculates a difference between the position of the similar image region and the position of the target block. Further, in the multi-viewpoint image depth value extraction method, the average value of the difference for each reference captured image calculated in the difference calculation step is calculated in the average parallax amount calculation step, and is set as the parallax amount in the target block. In this average parallax amount calculating step, the average parallax amount is obtained by excluding a difference corresponding to a similar image region whose similarity to the target block is lower than a predetermined value from the value for calculating the average value. After that, the multi-viewpoint image depth value extraction method converts the parallax amount calculated in the average parallax amount calculating step into the target block based on the predetermined focal length of the camera and the distance between the cameras in the depth value conversion step. Convert to the depth value of the corresponding subject.

The multi-viewpoint image depth value extraction method is a multi-viewpoint image depth value extraction method that extracts depth values for a multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged. Then, the block dividing unit divides the plurality of photographed images into blocks of a predetermined size, and the block search unit extracts the depth value of the photographed image divided in the block dividing step. A block search step for searching for a similar image region having the highest similarity of image features in a plurality of reference captured images captured by a camera adjacent to the camera that captured the captured image to which the target block belongs for each target block to be processed. And the position of the similar image area searched in the block search step and the target block by the difference calculating means. A difference calculating step for calculating a difference from the position of the reference image; and an average parallax amount calculating unit that calculates an average value of differences for each of the reference captured images calculated in the difference calculating step; The parallax amount calculated in the average parallax amount calculating step is set to a predetermined focal length of the camera and a distance between the cameras for each captured image by the average parallax amount calculating step and the depth value converting unit. Based on the depth value conversion step of converting the depth value of the subject corresponding to the target block, and peripheral block specifying means, the block over which the similar image area is straddled is specified as a peripheral block in the plurality of reference captured images The surrounding block specifying step, and the similar image region depth value calculating means, the surrounding block specifying step The similar image region depth is calculated by a similar image region depth value calculating step of calculating an average value of the depth values corresponding to the identified peripheral blocks and setting the depth value corresponding to the similar image region, and an average depth value calculating unit. An average depth value calculating step of calculating an average value of the depth value calculated in the value calculating step and the depth value of the target block and setting the depth value of the target block may be used.

At this time, in the multi-viewpoint image depth value extraction method, when calculating the average value of the depth values corresponding to the peripheral blocks in the similar image region depth value calculating step, the difference predetermined for the average value in the peripheral blocks is calculated. When a depth value having a larger difference corresponds to the average value, the average value may be calculated again by excluding the depth value (claim 5).

Further, the multi-viewpoint image depth value extraction method uses the similar image region when calculating the average value of the depth value calculated in the similar image region depth value calculation step and the depth value of the target block in the average depth value calculation step. The depth value calculated in the depth value calculation step may be calculated again by excluding depth values having a difference larger than a predetermined difference with respect to the average value (claim 6).

Further, the multi-viewpoint image depth value extraction program according to claim 7 , in order to extract a depth value for the multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged, The computer is configured to function as block dividing means, block searching means, difference calculating means, average parallax amount calculating means, and depth value converting means.

  In such a configuration, the multi-viewpoint image depth value extraction program divides a plurality of captured images into blocks having a predetermined size by the block dividing unit. Then, the multi-viewpoint image depth value extraction program uses a block search unit for each target block from which a depth value is to be extracted. In the photographed image, a similar image region having the highest image feature similarity is searched.

Then, the multi-viewpoint image depth value extraction program calculates the difference between the position of the similar image region and the position of the target block by the difference calculation means. Furthermore, the multi-viewpoint image depth value extraction program calculates an average value of differences for each reference captured image calculated by the difference calculation unit by the average parallax amount calculation unit, and sets the average value as the parallax amount in the target block. At this time, the average parallax amount calculating means excludes the difference corresponding to the similar image area whose similarity with the target block is lower than a predetermined value from the value for calculating the average value, and calculates the average parallax amount. Ask. Then, the multi-viewpoint image depth value extraction program converts the parallax amount calculated by the average parallax amount calculating unit to the target block based on the predetermined focal length of the camera and the distance between the cameras. Convert to the depth value of the corresponding subject.

In addition, the multi-viewpoint image depth value extracting program extracts a depth value for the multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged, the computer A block dividing unit that divides a captured image into blocks of a predetermined size, and for each target block from which the depth value of the captured image divided by the block dividing unit is to be extracted, the captured image to which the target block belongs Block search means for searching for a similar image area having the highest similarity of image features in a plurality of reference photographed images taken by cameras adjacent to the photographed camera, the position of the similar image area searched by the block search means, and the position Difference calculating means for calculating the difference with the position of the target block, the difference for each reference captured image calculated by the difference calculating means An average parallax amount calculating unit that calculates an average value and sets the parallax amount in the target block, and for each captured image, the parallax amount calculated by the average parallax amount calculating unit is set to a predetermined focal length of the camera and Based on the distance between the cameras, a depth value converting means for converting into a depth value of the subject corresponding to the target block, and in a plurality of the reference photographed images, a block spanned by the similar image region is specified as a peripheral block Peripheral block specifying means, a similar image area depth value calculating means for calculating an average value of the depth values corresponding to the peripheral blocks specified by the peripheral block specifying means, and making the depth value corresponding to the similar image area, this similarity The average value of the depth value calculated by the image area depth value calculating means and the depth value of the target block is calculated and used as the depth value of the target block. Hitoshi depth value calculating means may be configured to function as a.

At this time, the multi-viewpoint image depth value extraction program, when the similar image area depth value calculation means calculates the average value of the depth value corresponding to the peripheral block, the difference predetermined for the average value in the peripheral block When the depth value having a larger difference corresponds, the average value may be calculated again by excluding the depth value (claim 8).

Further, the multi-viewpoint image depth value extraction program allows the similar image region when the average depth value calculation unit calculates the average value of the depth value calculated by the similar image region depth value calculation unit and the depth value of the target block. The depth value calculated by the depth value calculating means may be configured to exclude the depth value having a difference larger than a predetermined difference with respect to the average value and calculate the average value again (claim 9).

The present invention has the following excellent effects.
According to the inventions described in claims 1, 4 and 7 , since the amount of parallax can be averaged between images taken by adjacent cameras, even if the corresponding points are not completely matched, Block matching errors are dispersed and reduced. Accordingly, the present invention can determine the depth value from the parallax amount with reduced error, and can accurately extract the depth value for the multi-viewpoint image of the subject from a plurality of captured images of the subject. .

In addition, according to the first, fourth, and seventh aspects of the present invention, it is possible to exclude the amount of parallax of similar image regions having greatly different degrees of similarity, and even when occlusion occurs, The amount of parallax can be calculated with high accuracy.

According to the inventions described in claims 2, 3, 5 , 6 , 8 , and 9, since the depth value of the similar image region can be averaged by the depth value associated with the peripheral block, The depth value can be obtained with reduced error. Accordingly, the present invention corrects the depth value of the target block from the depth value of the similar image area in which the error is reduced, and thus the accuracy of the depth value of the target block can be increased.

Further , according to the inventions of claims 2 , 5 and 8 , it is possible to exclude the depth values of the peripheral blocks having greatly different depth values, and even if occlusion occurs in the peripheral blocks, The depth value of the similar image area can be calculated with high accuracy. Thereby, the accuracy of the depth value of the target block can be increased.

Also, according to the invention as defined in claim 3,6,9, occlusion is more generated around the block, even if the depth value incorrect similar image areas are calculated, with the wrong depth values The depth value of the target block can be calculated by excluding the reference captured image. As a result, the depth value of the target block can be accurately extracted even when a plurality of occlusions occur in the captured image.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment: Configuration of Multi-Viewpoint Image Depth Value Extraction Device]
First, the configuration of the multi-viewpoint image depth value extraction device according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing the overall configuration of the multi-viewpoint image depth value extraction device according to the first embodiment of the present invention.

The multi-viewpoint image depth value extraction apparatus 1 extracts depth values for a multi-viewpoint image of a subject from a plurality of captured images obtained by capturing the same subject with a plurality of cameras arranged. The multi-viewpoint image depth value extracting apparatus 1 is connected to a plurality (three or more) of cameras C (..., C _N−1 , C _N , C _{N + 1} ,. The depth value of the subject is obtained using the captured image.

  The camera C outputs the captured image to the multi-viewpoint image depth value extraction device 1 in units of frames. The arrangement of the camera C is not limited as long as the distance from the camera C to the subject S is substantially equal between adjacent cameras. For example, in the case of extracting a depth value of a multi-viewpoint image used for generating a stereoscopic video, the cameras C may be arranged on a straight line at regular intervals in the horizontal direction facing the subject S. Further, for example, in the case of extracting the depth value of a multi-viewpoint image used for generating a free viewpoint video, the cameras C may be arranged on the ring at regular intervals around the subject S. Here, a description will be given with an example in which the cameras C are arranged on a straight line at regular intervals in the horizontal direction facing the subject S.

  Hereinafter, each configuration of the multi-viewpoint image depth value extraction device 1 will be described. Here, the multi-viewpoint image depth value extracting apparatus 1 includes a block dividing unit 10, a block searching unit 20, a parallax amount calculating unit 30, and a depth value converting unit 40.

The block dividing unit 10 divides a plurality of captured images captured by the camera C into blocks having a predetermined size for each captured image. This block serves as a reference when searching (block matching) a similar region between captured images, and has a size of, for example, 16 × 16 pixels. This block is also a reference for calculating the depth value of the subject S. Here, the block dividing means 10 accumulates the image inputted from the camera C (..., C _N−1 , C _N , C _{N + 1} ,...) In a frame unit in a storage means such as a memory not shown. The block position and size are output to the block search means 20.

The block search means 20 is a camera (for example, a camera C _N ) that has taken a captured image (hereinafter referred to as a reference captured image) to which a target block from which a depth value is to be extracted belongs in units of blocks divided by the block dividing means 10. In the plurality of photographed images (hereinafter referred to as reference photographed images) photographed by cameras adjacent to the camera (for example, the cameras C _N-1 and C _{N + 1} ), the similarity having the highest image feature similarity with the target block for each reference photographed image This is to search for an image area. Here, the block search means 20 includes a comparison target area selection means 21, a similarity calculation means 22, and a similar image area determination means 23.

  The comparison target region selection unit 21 sequentially selects a region (comparison target region) for comparing the target block and the image feature in the reference captured image for each target block from which the depth value in the standard captured image is to be extracted. To choose. Here, the comparison target area selection unit 21 sets a vicinity area of a block at the same image position as the target block in the reference captured image as a selection range, and sequentially shifts an area having the same size as the target block within the selection range, The comparison target areas are sequentially selected. Further, the comparison target area selection unit 21 sequentially outputs the selected comparison target areas to the similarity calculation unit 22.

  Here, the neighborhood area of the block to be selected by the comparison target area selection means 21 is an area expanded by several blocks vertically and horizontally, for example. In this way, by limiting the selection range to the neighborhood region, it is possible to prevent an erroneous block from being searched for in the reference captured image and to reduce the amount of calculation.

  The similarity calculation unit 22 calculates, for each target block, the similarity between the target block and the comparison target regions sequentially selected by the comparison target region selection unit 21. Here, the similarity calculation unit 22 calculates the similarity between the target block and the comparison target region based on the image feature. For example, the closer the distance between the color vectors is between the target block and the comparison target region, for example. The similarity is set to a high value. It should be noted that a general index may be used for the image similarity, and a color histogram, luminance, or a combination thereof may be used in addition to the color vector. In addition, as the distance, an absolute value of a value difference for each pixel in the block or a value obtained by adding a squared value can be used. The similarity calculated by the similarity calculation unit 22 is output to the similar image region determination unit 23.

The similar image region determining unit 23 determines a region (similar image region) similar to the target block based on the similarity calculated by the similarity calculating unit 22. Here, the similar image region determining means 23 determines the comparison target region having the highest similarity in the reference captured image as the similar image region for each target block. In addition, the similar image region determination unit 23 outputs the position of the target block, the position of the similar image region, and the similarity to the parallax amount calculation unit 30.
By configuring the block search means 20 in this way, a similar image region in a reference captured image captured by the adjacent camera C is searched for each target block.

  For each target block, the parallax amount calculation means 30 is based on the difference between the position of the target block and the position of the similar image area corresponding to the target block searched by the block search means 20. Is calculated. Here, the parallax amount calculation unit 30 includes a difference calculation unit 31 and an average parallax amount calculation unit 32.

  The difference calculation means 31 calculates the difference between the position of the target block and the position of the similar image area. Here, the difference calculation means 31 calculates a difference for each reference captured image based on the target block. The difference calculated by the difference calculation unit 31 is output to the average parallax amount calculation unit 32.

  The average parallax amount calculation unit 32 calculates the average value of the differences calculated by the difference calculation unit 31 as the parallax amount in the target block. In this way, by averaging a plurality of parallax amounts, even if a single parallax amount includes an error, the error is dispersed and the accuracy of the parallax amount can be increased.

  In addition, when calculating the average value of the parallax amount, the average parallax amount calculating unit 32, when the similarity of the similar image region of the reference captured image that is the target of calculating the average value is lower than a predetermined value, The difference between the position of the similar image region and the position of the target block is excluded from the value for calculating the average value. In this way, by removing a similar image region having a low similarity from the object for which the amount of parallax is obtained, for example, when there is a region (occlusion region) that is not captured by a certain camera C, the camera C captures the image. The occlusion area can be excluded from the object for which the amount of parallax is to be obtained, and an accurate amount of parallax can be obtained.

  When the cameras C are not arranged at equal intervals, the average parallax amount calculation means 32 calculates the average value after normalizing the parallax amount by a ratio according to the distance between the cameras C. In this case, the average parallax amount calculating means 32 associates the parallax amount at a predetermined standard camera interval with the target block.

For each captured image, the depth value converting unit 40 uses the parallax amount calculated by the parallax amount calculating unit 30 based on the predetermined focal length of the camera C and the distance between the cameras C, and the subject corresponding to the target block. Is converted into a depth value. That is, the depth value conversion means 40, when the distance between the cameras is L and the focal length of the camera C is f, the parallax amount B calculated in association with the target block of the captured image by the following equation (3). Is converted into a depth value D of the subject S in the target block.
D = fL / B (3) As described above, the depth value conversion means 40 can associate the depth value of the subject with each block in the plurality of captured images.

  By configuring the multi-viewpoint image depth value extraction device 1 as described above, the multi-viewpoint image depth value extraction device 1 can perform multiple viewpoints of a subject from a plurality of captured images obtained by photographing the same subject using a plurality of cameras arranged. The depth value for the image can be extracted. In addition, since the multi-viewpoint image depth value extraction device 1 calculates the parallax amount for each block based on the average value of the parallax amounts between the standard captured image and the plurality of reference captured images, the error is dispersed and the accuracy of the depth value is increased. Can be increased. Furthermore, when the multi-viewpoint image depth value extraction apparatus 1 calculates the amount of parallax by the average process, occlusion occurs because the similarity between blocks is lower than a predetermined value from the target of the average process. The depth value can be obtained accurately without being affected by the block.

  Note that the multi-viewpoint image depth value extraction apparatus 1 can be operated by a multi-viewpoint image depth value extraction program that causes a general computer to function as each of the above-described units. Further, the multi-viewpoint image depth value extraction program can be distributed via a communication line, or can be recorded and distributed on a recording medium such as a CD-ROM.

[First Embodiment: Operation of Multi-Viewpoint Image Depth Value Extraction Device]
Next, the operation of the multi-viewpoint image depth value extraction device according to the first embodiment of the present invention will be described with reference to FIG. FIG. 2 is a flowchart showing the operation of the multi-viewpoint image depth value extraction device according to the first embodiment of the present invention.

(Block division step)
First, the multi-viewpoint image depth value extracting apparatus 1 divides a plurality of captured images captured by the camera C into blocks having a predetermined size for each captured image by the block dividing unit 10 (step S1).

(Block search step)
Then, the multi-viewpoint image depth value extraction apparatus 1 uses the block search unit 20 to set one captured image as a reference captured image, and a camera (for example, camera C _N ) adjacent to the camera (for example, the camera C _N ) that captured the reference captured image. _In a plurality of reference captured images captured by ( _N−1 , C _{N + 1} ), a similar image region having the highest similarity of image features is searched for each block in the standard captured image.

  Specifically, the multi-viewpoint image depth value extraction device 1 uses the comparison target area selection unit 21 of the block search unit 20 to extract a block (target) from which the depth value in the standard captured image is extracted in the reference captured image. The area for comparing the block) and the image feature (comparison target area) is selected, and the similarity calculation unit 22 of the block search unit 20 calculates the similarity between the target block and the comparison target area (step S2).

  Here, the multi-viewpoint image depth value extraction apparatus 1 does not select the comparison target area for all the neighboring areas of the area corresponding to the target block by the comparison target area selection means 21 of the block search means 20 (in step S3). No), the comparison target region selection means 21 changes the comparison target region (step S4), and returns to step S2 to continue the operation.

  On the other hand, when the comparison target region selection unit 21 selects the comparison target regions for all the neighboring regions corresponding to the target block (Yes in step S3), the similar image region determination unit 23 of the block search unit 20 performs step S2. The comparison target region having the highest similarity calculated in step S5 is determined as a similar image region (step S5).

  Here, the multi-viewpoint image depth value extraction device 1 changes the similar captured image when the block search unit 20 has not determined similar image regions for all reference captured images (No in step S6) (step S7). Returning to step S2, the operation is continued.

(Parallax amount calculation step)
On the other hand, when the block search unit 20 determines similar image regions for all reference captured images (Yes in step S6), the multi-viewpoint image depth value extraction device 1 uses the parallax amount calculation unit 30 to calculate the parallax amount in the target block. calculate.

  Specifically, the multi-viewpoint image depth value extraction device 1 uses the difference calculation unit 31 of the parallax amount calculation unit 30 to calculate the difference between the position of the target block and the positions of the plurality of similar image regions determined in step S5. Calculate (step S8; difference calculation step).

  Then, the multi-viewpoint image depth value extraction device 1 calculates the average value of the differences calculated in step S8 as the parallax amount in the target block by the average parallax amount calculation unit 32 of the parallax amount calculation unit 30 (step S9; Average parallax amount calculating step). At this time, when the similarity to the target block in the similar image area is lower than a predetermined value, the average parallax amount calculating unit 32 calculates the difference between the position of the similar image area and the position of the target block. Exclude from the average value.

(Depth value conversion step)
Thereafter, the multi-viewpoint image depth value extraction apparatus 1 uses the depth value conversion unit 40 to calculate the parallax amount calculated in step S9 based on the predetermined focal length of the camera C and the distance between the cameras C. Is converted into the depth value of the subject corresponding to (step S10).

  When the depth values are not obtained for all the blocks in the reference captured image (No in step S11), the multi-viewpoint image depth value extraction device 1 changes the target block in the reference captured image (step S12). Returning to step S2, the operation is continued.

On the other hand, when the depth value is obtained for all the blocks in the reference photographed image (Yes in step S11) and the depth value is not obtained for other photographed images (No in step S13), the multi-viewpoint image depth value extracting device. 1 changes the reference captured image to another captured image (step S14), and returns to step S2 to continue the operation.
And when the depth value is calculated in all the captured images (Yes in step S13), the multi-viewpoint image depth value extraction device 1 ends the operation.

  When the camera C is a moving image camera, the multi-viewpoint image depth value extraction apparatus 1 sequentially inputs images in units of frame images and repeatedly executes the above operation. Thereby, the multi-viewpoint image depth value extraction device 1 can extract the depth value in units of frames even in the moving image video.

  With the above operation, the multi-viewpoint image depth value extraction apparatus 1 can extract the depth value for the multi-viewpoint image of the subject from the plurality of captured images obtained by capturing the same subject with a plurality of cameras arranged.

[Specific operation example of multi-viewpoint image depth value extraction apparatus 1]
Next, a specific operation example of the multi-viewpoint image depth value extraction device according to the first embodiment of the present invention will be described with reference to FIG. FIG. 3 is a schematic diagram for visually explaining the operation of the multi-viewpoint image depth value extraction device according to the first embodiment of the present invention.

3 (a) is a subject S plurality of cameras _{C (..., C N-1} , C N, C N + 1, ...) of the captured photographed images, the three cameras _{C N-1, C N,} The contents of the captured images M _N−1 , M _N , and M _{N + 1} captured by C _{N + 1} are shown.

First, the multi-viewpoint image depth value extraction apparatus 1 divides each captured image M _N−1 , M _N , M _{N + 1} into blocks having a predetermined size by the block dividing unit 10. FIG. 3 shows an example in which each image is divided into nine blocks.

The multi-viewpoint image depth value extraction apparatus 1 uses the block search unit 20 to capture one captured image as a reference captured image (here, M _N ) and a captured image captured by a camera adjacent to the camera that captured the reference captured image. As a reference captured image (here, M _N−1 , M _{N + 1} ), a region (similar image region) similar to the target block T _N in the standard captured image M _{N is included} in the reference captured image M _N−1 , M _{N + 1} . Search with. In FIG. 3 (a), similar image region _{R N-1} in the reference photographed image _{M N-1,} is similar image region _{R N + 1} in the reference photographed image _{M N + 1,} is searched as a region similar to the target block _{T N} Indicates the state.

Then, the multi-viewpoint image depth value extraction device 1 uses the parallax amount calculation unit 30 to calculate the difference between the position of the target block T _{N and} the positions of the similar image regions R _N−1 and R _{N + 1} (here, B _N−1 , B _{N + 1} ) is the parallax amount ((B _N−1 + B _{N + 1} ) / 2) of the target block T _N. When occlusion occurs in a block similar to the target block _TN as in the similar image region _{RN + 1} (here, a part of the corresponding block is hidden by a cylinder), the target block T _The similarity between _N and the similar image region _{RN + 1} is low. In this case, the parallax amount calculation means 30 employs the parallax amount B _N−1 with the similar image region R _N−1 as the parallax amount of the target block T _N.

Then, the multi-viewpoint image depth value extraction device 1 is in the reference photographed image M _N, sequentially, by changing the target block T _N, and calculates a parallax amount for all the blocks in the reference photographed image M _N. Then, the multi-viewpoint image depth value extraction apparatus 1 converts the parallax amount calculated for each block into the depth value by the depth value conversion means 40 (see the above-described formula (3)). Thereby, the multi-viewpoint image depth value extraction apparatus 1 can extract the depth value for each block in the reference captured image _MN .

Further, the multi-viewpoint image depth value extraction device 1 changes the reference captured image to another captured image as shown in FIG. FIG. 3B shows an example in which the captured image _MN-1 captured by the camera _CN-1 is changed to a reference captured image. As described above, the depth value can be extracted for each block in all the captured images by sequentially changing the reference captured image.

  As described above, the multi-viewpoint image depth value extraction device 1 obtains the parallax amount by averaging the difference between the position of the similar image region of the reference captured image captured by the adjacent camera and the position of the target block. Therefore, even if the corresponding points are different due to different cameras, the parallax error is reduced. Thereby, the depth value can be extracted with high accuracy.

[Second Embodiment: Configuration of Multi-Viewpoint Image Depth Value Extraction Device]
Next, the configuration of the multi-viewpoint image depth value extraction device according to the second embodiment of the present invention will be described with reference to FIG. FIG. 4 is a block diagram showing the overall configuration of the multi-viewpoint image depth value extraction device according to the second embodiment of the present invention.

  As shown in FIG. 4, the multi-viewpoint image depth value extracting apparatus 1B includes a block dividing unit 10, a block searching unit 20, a parallax amount calculating unit 30, a depth value converting unit 40, and a depth value correcting unit 50. Prepare. Since the configuration other than the depth value correction unit 50 is the same as that of the multi-viewpoint image depth value extraction device 1 described with reference to FIG.

  Similar to the multi-viewpoint image depth value extraction device 1 described with reference to FIG. 1, the multi-viewpoint image depth value extraction device 1B is for a multi-viewpoint image of a subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged. The depth value is extracted. Hereinafter, the depth value correcting means 50 having a configuration different from that of the multi-viewpoint image depth value extracting apparatus 1 will be described in detail.

  The depth value correcting unit 50 corrects the depth value for each block converted by the depth value converting unit 40 with reference to a captured image (reference captured image) captured by a camera other than the camera that captured the block. is there. Here, the depth value correcting unit 50 includes a peripheral block specifying unit 51, a similar image region depth value calculating unit 52, and an average depth value calculating unit 53.

  The peripheral block specifying means 51 is for specifying, as a peripheral block, a block in which a similar image region is straddled in a plurality of reference captured images. The peripheral block specifying unit 51 is similar to the target block in the reference captured image by shifting the position of the block (corresponding block) in the reference captured image corresponding to the target block by the amount of parallax between the target block and the corresponding block. The position of the similar image area is obtained, and the peripheral block over which the similar image area extends is specified.

For example, when the depth value of the target block is D (= fL / B [see the above equation (3)]), the peripheral block specifying unit 51 indicates the position of the corresponding block from the following equation (4). The parallax amount B is obtained by inverse conversion, and the position of the similar image region is specified by shifting the corresponding block by the parallax amount B.
B = fL / D (4) formula

  Then, the peripheral block specifying unit 51 outputs the blocks around the similar image region to the similar image region depth value calculating unit 52. When the cameras C are not arranged at equal intervals, the peripheral block specifying unit 51 is based on a value obtained by converting the parallax amount at the standard camera interval associated with the target block into the parallax amount at the camera interval. The position of the similar image area is specified.

  Here, the peripheral block specifying unit 51 specifies the position of the similar image region by shifting the block (corresponding block) of the reference captured image corresponding to the position of the target block by the amount of parallax. The position of the similar image area determined by the similar image area determining unit 23 may be held in a storage unit (not shown) and used.

  The similar image region depth value calculating unit 52 calculates an average value of the depth values corresponding to the blocks (peripheral blocks) specified by the peripheral block specifying unit 51 in the reference photographed image, and the depth value corresponding to the similar image region. It is what.

  The similar image area depth value calculating means 52 calculates the depth value of the similar image area by calculating the average value of the depth values corresponding to the peripheral blocks over which the similar image area is straddled. Even if an error is included in the error, the error is dispersed, and a depth value with high accuracy can be obtained.

The similar image region depth value calculation means 52 is a depth value in which the difference between the depth value of the similar image region and the depth value of each peripheral block used for calculating the depth value is larger than a predetermined value. For the peripheral blocks having “”, it is desirable to calculate the average value again by excluding the value of the depth value and use it as the depth value of the similar image region. Thereby, the similar image region depth value calculation means 52 excludes the depth value of the peripheral block in which the occlusion occurs when the occlusion occurs in the peripheral block across the similar image region and the correct depth value does not correspond. Therefore, since the average value is obtained, the depth value of the similar image region with higher accuracy can be obtained.
The depth values of the plurality of similar image areas corresponding to the respective target blocks thus obtained are output to the average depth value calculation unit 53.

  The average depth value calculating unit 53 calculates an average value of the depth values of the plurality of similar image regions corresponding to the target block calculated by the similar image region depth value calculating unit 52 and the depth value of the target block, and This is the depth value of the target block. In this way, by averaging the depth values corresponding to a plurality of similar blocks, even if the depth value includes an error, the error is dispersed and the depth value is highly accurate. Can be requested.

  The average depth value calculation means 53 is a depth value in which the difference between the depth value of the target block obtained by averaging with the depth value of the similar image area and the depth value of the similar image area is larger than a predetermined value. For a similar image region having a similar image region, it is desirable to calculate the average value again by excluding the reference captured image including the depth value value of the similar image region, and to obtain the depth value of the target block. Thereby, the average depth value calculation means 53 is a case where the similar image area depth value calculation means 52 has a plurality of occlusions in the similar image area and associates an incorrect depth value with the similar image area. Since the average value is obtained by excluding the reference photographed image including the erroneous depth value, the depth value of the similar image region with higher accuracy can be obtained.

  As described above, the depth value correcting unit 50 calculates the depth value of the similar image region by averaging the depth values of the blocks over which the similar image region in the reference photographed image spans, and calculates a plurality of depth values for each target block. The depth value of the target block is corrected by averaging the depth values of the similar image regions of the reference captured image. As a result, the depth value correcting means 50 can improve the accuracy of the depth value due to the error variance even when there is an error in the depth value of the target block converted by the depth value converting means 40.

  In addition, the depth value correction | amendment means 50 does not specifically limit the number of the reference picked-up images containing the similar image area | region for correct | amending the depth value of an object block. For example, the depth value correcting unit 50 corrects the depth value of the target block using the depth values of two similar image areas in two reference captured images captured by cameras adjacent to the camera that captured the reference captured image. Alternatively, the reference captured image may be further increased, and the depth value of the target block may be corrected using the similar image regions of the three or more reference captured images.

  As described above, by configuring the multi-viewpoint image depth value extraction device 1B, the multi-viewpoint image depth value extraction device 1B can perform multi-viewpoints of subjects from a plurality of captured images obtained by photographing the same subject using a plurality of cameras arranged. The depth value for the image can be extracted. In addition, since the multi-viewpoint image depth value extraction device 1B averages the depth values of blocks similar to the target block from captured images taken by a plurality of cameras, even when a plurality of occlusions occur, The depth value can be obtained with high accuracy.

  The multi-viewpoint image depth value extraction apparatus 1B can operate a general computer by a multi-viewpoint image depth value extraction program that functions as each of the above-described units. Further, the multi-viewpoint image depth value extraction program can be distributed via a communication line, or can be recorded and distributed on a recording medium such as a CD-ROM.

[Second Embodiment: Operation of Multi-Viewpoint Image Depth Value Extraction Device]
Next, the operation of the multi-viewpoint image depth value extraction device according to the second embodiment of the present invention will be described with reference to FIG. FIG. 5 is a flowchart showing the operation of the multi-viewpoint image depth value extraction device according to the second embodiment of the present invention. The operations in steps S1 to S14 in FIG. 5 are the same as the operations of the multi-viewpoint image depth value extraction apparatus 1 (FIG. 1) described in FIG. Here, the operation after step S14 will be described.

(Depth value correction step)
After the operation of step S14, the multi-viewpoint image depth value extraction device 1B captures the depth value for each block converted by the depth value conversion unit 40 by the depth value correction unit 50 with a camera other than the camera that captured the block. Correction is performed with reference to the captured image (reference captured image).

  Specifically, the multi-viewpoint image depth value extraction device 1B specifies, as a peripheral block, a block that spans similar image regions in a plurality of reference photographed images by the peripheral block specifying unit 51 of the depth value correcting unit 50 (step). S15: peripheral block specifying step).

  Then, the multi-viewpoint image depth value extraction device 1B corresponds to the peripheral blocks in the reference photographed image spanned by the similar image region specified in step S15 by the similar image region depth value calculating unit 52 of the depth value correcting unit 50. An average value of the depth values is calculated and set as a depth value corresponding to the similar image region (step S16; similar image region depth value calculating step).

  It should be noted that the depth value of the peripheral block in which the difference between the average value of the depth value of the similar image region and the depth value of each peripheral block used for calculating the average value is larger than a predetermined value (threshold value). If it exists (Yes in step S17), the similar image area depth value calculation means 52 calculates the average value again excluding the depth values of the peripheral blocks where the difference is large, and the depth value corresponding to the similar image area (Step S18). Then, the operation proceeds to step S19.

  On the other hand, when the difference between the average value of the depth value of the similar image region and the depth value of each peripheral block used for calculating the average value is equal to or less than a predetermined value (threshold value) (No in step S17). ), The multi-viewpoint image depth value extraction apparatus 1B uses the average depth value calculation unit 53 of the depth value correction unit 50 to calculate similar image regions of a plurality of reference captured images corresponding to the target block calculated in step S16 or step S18. An average value of the depth value and the depth value of the target block is calculated and set as the depth value of the target block (step S19; average depth value calculation step).

  Note that the depth value of the similar image region in which the difference between the average value calculated in step S19 and the depth value of the similar image region used for calculating the average value is larger than a predetermined value (threshold value). If there is a reference captured image that includes the image (Yes in step S20), the average depth value calculation unit 53 calculates the average value again, excluding the reference captured image that includes the depth value of the similar image region where the difference is large. The depth value corresponding to the target block is set (step S21).

  And when the correction value of depth value is not acquired about all the blocks in a reference | standard imaging | photography image (it is No at step S22), the multiview image depth value extracting device 1B changes the object block in a reference | standard imaging | photography image ( Step S23), returning to step S15, the operation is continued.

On the other hand, depth value correction values are obtained for all the blocks in the reference captured image (Yes in step S22), and depth value correction values are not obtained for other captured images (No in step S24). The viewpoint image depth value extraction device 1B changes the reference captured image to another captured image (step S25), returns to step S15, and continues the operation.
And when the correction value of depth value is calculated in all the picked-up images (Yes in step S24), the multi-viewpoint image depth value extraction device 1B ends the operation.

  Through the above operation, the multi-viewpoint image depth value extraction device 1B corrects the depth value of the block extracted by the multi-viewpoint image depth value extraction device 1 described in FIG. 2 with reference to the similar image region of the reference captured image. The depth value of the block can be extracted with higher accuracy.

[Specific Operation Example of Multi-viewpoint Image Depth Value Extracting Device 1B]
Next, a specific operation example of the multi-viewpoint image depth value extraction device according to the second embodiment of the present invention will be described with reference to FIG. FIG. 6 is a schematic diagram for visually explaining the operation of the multi-viewpoint image depth value extracting device according to the second embodiment of the present invention. The operation for extracting the depth value for each block described in FIG. 3 is the same in the multi-viewpoint image depth value extracting apparatus 1B. Here, only the operation for correcting the depth value of the block will be described. .

In FIG. 6, three cameras C _N−1 , C _N , and C _{N + 1} out of the captured images obtained by shooting the subject S with a plurality of cameras C (..., C _N−1 , C _N , C _{N + 1} ,. The contents of the captured images M _N−1 , M _N , and M _{N + 1} are shown. In addition, _MN is a standard captured image, and M _N−1 and M _{N + 1} are reference captured images.

First, the multi-viewpoint image depth value extracting apparatus 1B, the peripheral block specifying means 51 of the depth value correction unit 50, based on the depth value associated with the target block T _N in the reference photographed image M _N to the parallax amount By performing inverse transformation, the position of the similar image region is specified in the reference photographed images M _N−1 and M _{N + 1} . Here, a state in which similar image regions R _N−1 and R _{N + 1} corresponding to the target block T _N are specified in the reference photographed images M _N−1 and M _{N + 1} is shown. Further, the peripheral block specifying means 51 specifies blocks (peripheral blocks) over which the similar image regions R _N−1 and R _{N + 1} straddle in the reference photographed images M _N−1 and M _{N + 1} , respectively.

Then, the multi-viewpoint image depth value extraction apparatus 1B uses the similar image region depth value calculation unit 52 of the depth value correction unit 50 to calculate the average value of the depth values corresponding to the peripheral blocks to the similar image regions R _N−1 and R _{N + 1.} Calculated as the depth value of.

Here, around the block similar image areas in the reference photographed image _{M N-1 R N-1} is across the T1-T4, the reference photographed image _{M N +} neighboring block similar image region _{R N + 1} extends over within ₁ T5~ T8 is indicated. It is assumed that depth values D1 to D8 are associated with the peripheral blocks T1 to T8, respectively. Then, the similar image region depth value calculating unit 52 calculates the depth value Da of the similar image region RN _-1 as an average value ((D1 + D2 + D3 + D4) / 4) of the depth values of the peripheral blocks T1 to T4. The depth value Db of RN _{+ 1} is calculated as the average value ((D5 + D6 + D7 + D8) / 4) of the depth values of the peripheral blocks T5 to T8.

Note that the similar image region depth value calculation unit 52 generates occlusion (in this case, a part of the corresponding block is hidden by a cylinder) as shown in the peripheral block T7 in the reference captured image _{MN + 1} , for example. If the difference between the depth value of the peripheral block T7 and the depth value Db of the similar image region RN _{+ 1} is larger than a predetermined value, the depth value of the peripheral block T7 is excluded and the similar image region R The depth value Db of _{N + 1} is calculated as an average value ((D5 + D6 + D8) / 3) of the depth values of the peripheral blocks T5, T6, T8.

Then, the multi-viewpoint image depth value extracting apparatus 1B, the average depth value calculating means 53 of the depth value correction unit 50, a depth value Da of the similar image regions R _N-1, and the depth value Db of the similar image regions R _{N + 1,} calculated average value of the depth value D of the target block _{T N} a ((Da + Db + D) / 3), to correct the depth value D of the target block _{T N.} When the depth values Da and Db of the similar image regions R _N−1 and R _{N + 1} are larger than a predetermined value with respect to the depth value D of the target block _TN calculated as an average value, It is desirable to obtain the average value again by excluding the depth value of the similar image region.

  As described above, since the multi-viewpoint image depth value extraction device 1B corrects the depth value of the target block with the average value of the depth value of the similar image region of the other reference captured images, the captured cameras are different. Even if the corresponding points are different, the error of the depth value is reduced, and the depth value can be extracted with high accuracy.

1 is a block diagram illustrating an overall configuration of a multi-viewpoint image depth value extraction device according to a first embodiment of the present invention. It is a flowchart which shows operation | movement of the multi-viewpoint image depth value extraction apparatus which concerns on 1st Embodiment of this invention. It is a schematic diagram for demonstrating visually operation | movement of the multi-viewpoint image depth value extraction apparatus which concerns on 1st Embodiment of this invention. It is a block diagram which shows the whole structure of the multiview image depth value extraction apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the multiview image depth value extraction apparatus which concerns on 2nd Embodiment of this invention. It is a schematic diagram for demonstrating visually operation | movement of the multi-viewpoint image depth value extraction apparatus which concerns on 2nd Embodiment of this invention. It is explanatory drawing for demonstrating the method of calculating | requiring a parallax by the conventional block matching.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 1B Multi-viewpoint image depth value extraction apparatus 10 Block division means 20 Block search means 21 Comparison object area selection means 22 Similarity calculation means 23 Similar image area determination means 30 Parallax amount calculation means 31 Difference calculation means 32 Average parallax amount calculation means 40 Depth Value Conversion Unit 50 Depth Value Correction Unit 51 Peripheral Block Identification Unit 52 Similar Image Area Depth Value Calculation Unit 53 Average Depth Value Calculation Unit

Claims (9)

A multi-viewpoint image depth value extraction device that extracts a depth value for a multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged,
Block dividing means for dividing the plurality of photographed images into blocks of a predetermined size;
In a plurality of reference captured images captured by a camera adjacent to the camera that captured the captured image to which the target block belongs, for each target block from which the depth value of the captured image divided by the block dividing unit is extracted, Block search means for searching for a similar image region having the highest similarity of image features;
A difference calculating means for calculating a difference between the position of the similar image area searched by the block searching means and the position of the target block;
An average parallax amount calculating unit that calculates an average value of the difference for each reference captured image calculated by the difference calculating unit and sets the parallax amount in the target block;
For each of the captured images, the parallax amount calculated by the average parallax amount calculating unit is determined based on the predetermined focal length of the camera and the distance between the cameras, and the depth value of the subject corresponding to the target block. and a depth value converting means for converting,
The average parallax amount calculating means includes
When the similarity between the similar image area of the reference captured image that is the target for calculating the average value and the target block is lower than a predetermined value, the difference between the position of the similar image area and the position of the target block is calculated. A multi-viewpoint image depth value extracting apparatus, characterized in that the average value is calculated by excluding the parallax amount . A multi-viewpoint image depth value extraction device that extracts a depth value for a multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged,
Block dividing means for dividing the plurality of photographed images into blocks of a predetermined size;
In a plurality of reference captured images captured by a camera adjacent to the camera that captured the captured image to which the target block belongs, for each target block from which the depth value of the captured image divided by the block dividing unit is extracted, Block search means for searching for a similar image region having the highest similarity of image features;
A difference calculating means for calculating a difference between the position of the similar image area searched by the block searching means and the position of the target block;
An average parallax amount calculating unit that calculates an average value of the difference for each reference captured image calculated by the difference calculating unit and sets the parallax amount in the target block;
For each of the captured images, the parallax amount calculated by the average parallax amount calculating unit is determined based on the predetermined focal length of the camera and the distance between the cameras, and the depth value of the subject corresponding to the target block. Depth value conversion means for converting to
In a plurality of the reference photographed images, peripheral block specifying means for specifying a block across the similar image region as a peripheral block;
A similar image area depth value calculating means for calculating an average value of the depth values corresponding to the peripheral blocks specified by the peripheral block specifying means, and making the depth value corresponding to the similar image area;
An average depth value calculating unit that calculates an average value of the depth value calculated by the similar image region depth value calculating unit and the depth value of the target block, and sets the depth value of the target block;
The similar image region depth value calculating means includes:
When calculating an average value of the depth values corresponding to the peripheral blocks, if the depth value corresponding to the peripheral block is larger than a predetermined difference with respect to the average value, the depth value is set. A multi-viewpoint image depth value extraction apparatus, characterized in that the average value is calculated again by excluding it . A multi-viewpoint image depth value extraction device that extracts a depth value for a multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged,
Block dividing means for dividing the plurality of photographed images into blocks of a predetermined size;
In a plurality of reference captured images captured by a camera adjacent to the camera that captured the captured image to which the target block belongs, for each target block from which the depth value of the captured image divided by the block dividing unit is extracted, Block search means for searching for a similar image region having the highest similarity of image features;
A difference calculating means for calculating a difference between the position of the similar image area searched by the block searching means and the position of the target block;
An average parallax amount calculating unit that calculates an average value of the difference for each reference captured image calculated by the difference calculating unit and sets the parallax amount in the target block;
For each of the captured images, the parallax amount calculated by the average parallax amount calculating unit is determined based on the predetermined focal length of the camera and the distance between the cameras, and the depth value of the subject corresponding to the target block. Depth value conversion means for converting to
In a plurality of the reference photographed images, peripheral block specifying means for specifying a block across the similar image region as a peripheral block;
A similar image area depth value calculating means for calculating an average value of the depth values corresponding to the peripheral blocks specified by the peripheral block specifying means, and making the depth value corresponding to the similar image area;
An average depth value calculating unit that calculates an average value of the depth value calculated by the similar image region depth value calculating unit and the depth value of the target block, and sets the depth value of the target block;
The average depth value calculating means is:
When calculating the average value of the depth value calculated by the similar image region depth value calculating unit and the depth value of the target block, the average value is calculated from the depth value calculated by the similar image region depth value calculating unit. A multi-viewpoint image depth value extracting apparatus, wherein a depth value having a difference larger than a predetermined difference is excluded and an average value is calculated again . A multi-viewpoint image depth value extraction method for extracting depth values for a multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged,
The block dividing means, a block dividing step of dividing the block size that defines a plurality of captured images in advance,
For each target block from which the depth value of the captured image divided in the block dividing step is extracted by the block search unit, a plurality of images captured by a camera adjacent to the camera that captured the captured image to which the target block belongs are captured. A block search step for searching for a similar image region having the highest similarity of the image features in the reference photographed image;
A difference calculating step of calculating a difference between the position of the similar image area searched in the block searching step and the position of the target block by a difference calculating means;
An average parallax amount calculating unit that calculates an average value of differences for each of the reference captured images calculated in the difference calculating step and sets the parallax amount in the target block;
For each captured image, the parallax amount calculated in the average parallax amount calculating step corresponds to the target block based on a predetermined focal length of the camera and a distance between the cameras by a depth value conversion unit. a depth value conversion step of converting the depth value of the subject which, only including,
In the average parallax amount calculating step, the average parallax amount calculating means includes:
When the similarity between the similar image area of the reference captured image that is the target for calculating the average value and the target block is lower than a predetermined value, the difference between the position of the similar image area and the position of the target block is calculated. A multi-viewpoint image depth value extraction method, characterized in that the average value is calculated by excluding the parallax amount . A multi-viewpoint image depth value extraction method for extracting depth values for a multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged,
The block dividing means, a block dividing step of dividing the block size that defines a plurality of captured images in advance,
For each target block from which the depth value of the captured image divided in the block dividing step is extracted by the block search unit, a plurality of images captured by a camera adjacent to the camera that captured the captured image to which the target block belongs are captured. A block search step for searching for a similar image region having the highest similarity of the image features in the reference photographed image;
A difference calculating step of calculating a difference between the position of the similar image area searched in the block searching step and the position of the target block by a difference calculating means;
An average parallax amount calculating unit that calculates an average value of differences for each of the reference captured images calculated in the difference calculating step and sets the parallax amount in the target block;
For each captured image, the parallax amount calculated in the average parallax amount calculating step corresponds to the target block based on a predetermined focal length of the camera and a distance between the cameras by a depth value conversion unit. A depth value conversion step for converting the depth value of the subject to
A peripheral block specifying step of specifying, as a peripheral block, a block across the similar image regions in the plurality of reference captured images by a peripheral block specifying unit;
Similar image region depth value calculation means calculates an average value of the depth values corresponding to the peripheral blocks specified in the peripheral block specifying step, and calculates a similar image region depth value as a depth value corresponding to the similar image region Steps,
An average depth value calculating unit calculates an average value of the depth value calculated in the similar image region depth value calculating step and the depth value of the target block, and sets the depth value of the target block as an average depth value calculating step; Including,
In the similar image region depth value calculating step, the similar image region depth value calculating means includes:
When calculating an average value of the depth values corresponding to the peripheral blocks, if the depth value corresponding to the peripheral block is larger than a predetermined difference with respect to the average value, the depth value is set. A multi-viewpoint image depth value extraction method, characterized by excluding the average value and excluding the average value . A multi-viewpoint image depth value extraction method for extracting depth values for a multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged,
The block dividing means, a block dividing step of dividing the block size that defines a plurality of captured images in advance,
For each target block from which the depth value of the captured image divided in the block dividing step is extracted by the block search unit, a plurality of images captured by a camera adjacent to the camera that captured the captured image to which the target block belongs are captured. A block search step for searching for a similar image region having the highest similarity of the image features in the reference photographed image;
A difference calculating step of calculating a difference between the position of the similar image area searched in the block searching step and the position of the target block by a difference calculating means;
An average parallax amount calculating unit that calculates an average value of differences for each of the reference captured images calculated in the difference calculating step and sets the parallax amount in the target block;
For each captured image, the parallax amount calculated in the average parallax amount calculating step corresponds to the target block based on a predetermined focal length of the camera and a distance between the cameras by a depth value conversion unit. A depth value conversion step for converting the depth value of the subject to
A peripheral block specifying step of specifying, as a peripheral block, a block across the similar image regions in the plurality of reference captured images by a peripheral block specifying unit;
Similar image region depth value calculation means calculates an average value of the depth values corresponding to the peripheral blocks specified in the peripheral block specifying step, and calculates a similar image region depth value as a depth value corresponding to the similar image region Steps,
An average depth value calculating unit calculates an average value of the depth value calculated in the similar image region depth value calculating step and the depth value of the target block, and sets the depth value of the target block as an average depth value calculating step; Including,
In the average depth value calculating step, the average depth value calculating means includes:
When calculating the average value of the depth value calculated in the similar image region depth value calculating step and the depth value of the target block, the average value is calculated from the depth value calculated in the similar image region depth value calculating step. A multi-viewpoint image depth value extraction method, wherein a depth value having a difference larger than a predetermined difference is excluded and an average value is calculated again . In order to extract a depth value for a multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged,
Block dividing means for dividing the plurality of captured images into blocks of a predetermined size;
In a plurality of reference captured images captured by a camera adjacent to the camera that captured the captured image to which the target block belongs, for each target block from which the depth value of the captured image divided by the block dividing unit is extracted, Block search means for searching for a similar image region having the highest similarity of image features;
A difference calculating means for calculating a difference between the position of the similar image area searched by the block searching means and the position of the target block;
An average parallax amount calculating unit that calculates an average value of the difference for each reference captured image calculated by the difference calculating unit and sets the parallax amount in the target block;
For each of the captured images, the parallax amount calculated by the average parallax amount calculating unit is determined based on the predetermined focal length of the camera and the distance between the cameras, and the depth value of the subject corresponding to the target block. depth value converting means for converting, to function as,
The average parallax amount calculating means includes
When the similarity between the similar image area of the reference captured image that is the target for calculating the average value and the target block is lower than a predetermined value, the difference between the position of the similar image area and the position of the target block is calculated. A multi-viewpoint image depth value extraction program characterized in that the average value is excluded and calculated as the parallax amount . In order to extract a depth value for a multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged,
Block dividing means for dividing the plurality of captured images into blocks of a predetermined size;
In a plurality of reference captured images captured by a camera adjacent to the camera that captured the captured image to which the target block belongs, for each target block from which the depth value of the captured image divided by the block dividing unit is extracted, Block search means for searching for a similar image region having the highest similarity of image features;
A difference calculating means for calculating a difference between the position of the similar image area searched by the block searching means and the position of the target block;
An average parallax amount calculating unit that calculates an average value of the difference for each reference captured image calculated by the difference calculating unit and sets the parallax amount in the target block;
For each of the captured images, the parallax amount calculated by the average parallax amount calculating unit is determined based on the predetermined focal length of the camera and the distance between the cameras, and the depth value of the subject corresponding to the target block. Depth value conversion means for converting to
In a plurality of the reference photographed images, peripheral block specifying means for specifying a block across the similar image region as a peripheral block,
A similar image region depth value calculating unit that calculates an average value of the depth values corresponding to the peripheral blocks specified by the peripheral block specifying unit and sets the depth value corresponding to the similar image region;
Calculating an average value of the depth value calculated by the similar image region depth value calculating unit and the depth value of the target block, and functioning as an average depth value calculating unit as a depth value of the target block;
The similar image region depth value calculating means includes:
When calculating an average value of the depth values corresponding to the peripheral blocks, if the depth value corresponding to the peripheral block is larger than a predetermined difference with respect to the average value, the depth value is set. A multi-viewpoint image depth value extraction program characterized by excluding the average value and excluding the average value . In order to extract a depth value for a multi-viewpoint image of the subject from a plurality of captured images obtained by photographing the same subject with a plurality of cameras arranged,
Block dividing means for dividing the plurality of captured images into blocks of a predetermined size;
In a plurality of reference captured images captured by a camera adjacent to the camera that captured the captured image to which the target block belongs, for each target block from which the depth value of the captured image divided by the block dividing unit is extracted, Block search means for searching for a similar image region having the highest similarity of image features;
A difference calculating means for calculating a difference between the position of the similar image area searched by the block searching means and the position of the target block;
An average parallax amount calculating unit that calculates an average value of the difference for each reference captured image calculated by the difference calculating unit and sets the parallax amount in the target block;
For each of the captured images, the parallax amount calculated by the average parallax amount calculating unit is determined based on the predetermined focal length of the camera and the distance between the cameras, and the depth value of the subject corresponding to the target block. Depth value conversion means for converting to
In a plurality of the reference photographed images, peripheral block specifying means for specifying a block across the similar image region as a peripheral block,
A similar image region depth value calculating unit that calculates an average value of the depth values corresponding to the peripheral blocks specified by the peripheral block specifying unit and sets the depth value corresponding to the similar image region;
Calculating an average value of the depth value calculated by the similar image region depth value calculating unit and the depth value of the target block, and functioning as an average depth value calculating unit as a depth value of the target block;
The average depth value calculating means is:
When calculating the average value of the depth value calculated by the similar image region depth value calculating unit and the depth value of the target block, the average value is calculated from the depth value calculated by the similar image region depth value calculating unit. A multi-viewpoint image depth value extraction program which calculates a mean value again by excluding depth values having a difference larger than a predetermined difference .

Images