JP2005165969A - Image processor and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method and a device which perform highly precise three-dimensional information estimation and perform image division closer to the original shape of an object. <P>SOLUTION: The method of three-dimensional information estimation and the device for performing the same divide an image used as reference into small areas, estimate parallax of boundary lines of the small areas by a stereo method of area base, analyze frequency distribution of the parallax of the boundary lines estimated in each area and estimate the parallax of the areas. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image processing method and apparatus for estimating three-dimensional information of a subject based on the correspondence between a plurality of images taken from different viewpoints, and the estimated three-dimensional information and image color information. The present invention relates to an image processing method and apparatus for performing image division based on the above.

  Conventionally, as a method for estimating 3D information of a target object, parallax is estimated from the correspondence between multiple images taken from different viewpoints, and 3D information is estimated from the parallax information based on the principle of triangulation Stereo methods are known. As the stereo method, a feature-based stereo method and a region-based stereo method are generally known.

  The feature-based stereo method is a method of extracting features such as edges from a plurality of images and obtaining the correspondence between the features. Features used include edge points and edge segments. While this method has the advantage of being relatively resistant to luminance fluctuations between images, many similar features are likely to occur when the image density is complex, and miscorrespondence is likely to occur, or features between multiple images that require correspondence There is a problem that the accuracy of the correspondence search is reduced when are not extracted equally. In addition, since only correspondence between features is required, when 3D information of the entire image is required, it is necessary to interpolate the parallax from the feature where the parallax is estimated to other regions. In order to extract features necessary for restoring the three-dimensional information, it is also necessary to strictly set the feature extraction processing.

  One example of a feature-based stereo method is Japanese Patent No. 2843034. In this method, a region in an image of a complex object having many textures is detected as a texture region, and the disparity is estimated by associating segments constituting the boundary of the texture region. Based on the disparity of the boundary The parallax within the area is obtained. When this method is used, it is possible to accurately estimate the parallax even for a complex region having many textures, which is difficult with the feature-based stereo method. However, in the case where the boundary line of the region is generated by an object having a different distance, the parallax of the different object is propagated in the region, and the accuracy of the parallax estimation is lowered. In addition, there remains a problem in the feature-based stereo method in which the parallax estimation is sparse in a region where the density changes smoothly.

  The region-based stereo method is a method of searching for an area most similar to a small area in a reference image in an image and obtaining a correspondence relationship. Compared with the feature-based stereo method, the parallax can be estimated in a region where the gray value of the image changes smoothly, and dense three-dimensional information can be obtained. However, a correspondence relationship cannot be obtained for a region where the gray value of the image hardly changes, and erroneous correspondence is likely to occur. In addition, there is a problem that the estimated parallax boundary and the contour of the object are shifted at the boundary of the object at different distances, and the three-dimensional information in the contour portion of the object cannot be estimated due to occlusion.

  One method for improving the problems of the conventional region-based stereo method is Japanese Patent Laid-Open No. 10-191396. In this method, the clustering result of the image is used to detect a part with less image features where the parallax estimation becomes unstable or a part with low reliability of the parallax estimation such as an occlusion area, and the parallax of that part is detected by the surrounding same cluster. Interpolation is performed from the parallax in the pixel to improve the deviation between the object outline and the parallax boundary. However, the calculation is redundant because an unreliable area is detected after the parallax estimation is performed, and when the background area has few image features, the reliability of the parallax estimation is not necessarily reduced in the object contour portion. However, there is a problem that the parallax cannot be interpolated and there is a case where the boundary between the object outline and the parallax may be shifted.

  In the conventional region-based stereo method, an erroneous correspondence occurs in an area where the change in image density is extremely small, the parallax cannot be estimated in the occlusion area, and the object contour line at the boundary between objects with different distances. In order to improve the problem that the parallax boundary is shifted, the reference image is divided into small areas, and the parallax of the boundary line of the small area is estimated by the area-based stereo method, and the boundary estimated in each area Disclosed is an image processing method and apparatus capable of estimating three-dimensional information with high accuracy by analyzing a frequency distribution of parallax of a line and estimating a parallax of a region. Further, the present invention is an image division and integration method in which an image is divided into initial regions and the divided small regions are sequentially integrated, and the three-dimensional information estimated by the method is used as a feature amount for integration. An image processing method and apparatus capable of dividing an image closer to the original shape of the subject are provided.

  In order to achieve the above object, a first three-dimensional information extraction apparatus of the present invention includes an image input means for inputting a plurality of image data taken at different viewpoint positions, and image data input by the image input means. Input by the image input means, area dividing means for dividing one reference image data into small areas, area boundary detecting means for detecting a boundary line of each small area divided by the area dividing means, The correspondence between a plurality of points on the boundary line of each small region detected by the region boundary line detection means is determined at least between the reference image of the plurality of images and another image, and a plurality of points on the boundary line of each small region are obtained. Parallax amount estimation means for estimating the amount of parallax at the point, and parallax amount frequency distribution creation means for creating a frequency distribution of the amount of parallax at a plurality of points on the boundary estimated by the parallax amount estimation means, for each small region, The The disparity amount frequency distribution analyzing means for analyzing the frequency distribution of the disparity amount of each small area created by the difference amount frequency distribution creating means, and the disparity of each small area based on the result analyzed by the disparity amount frequency distribution analyzing means It comprises an area parallax amount estimation means for estimating an amount, and a three-dimensional information calculation means for calculating three-dimensional information based on the parallax amount of each small area estimated by the area parallax amount estimation means.

  The second three-dimensional information extraction apparatus of the present invention includes an image input means for inputting a plurality of image data photographed at different viewpoint positions, and one reference among the image data input by the image input means. Area dividing means for dividing the image data into small areas; area boundary detecting means for detecting the boundary lines of the respective small areas divided by the area dividing means; and a plurality of images input by the image input means The correspondence between a plurality of points on the boundary line of each small region detected by at least the region boundary line detection means between the reference image and the other image is obtained, and the parallax amounts of the plurality of points on the boundary line of each small region are calculated. A parallax amount estimating means for estimating, a parallax amount frequency distribution creating means for creating a frequency distribution of parallax amounts of a plurality of points on a boundary line estimated by the parallax amount estimating means for each small region, and the parallax amount frequency distribution creating A parallax amount frequency distribution analyzing unit for analyzing the frequency distribution of the parallax amount of each small region created by the stage, and a region for estimating the parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analyzing unit A parallax amount estimating unit; a parallax amount frequency distribution correcting unit that corrects a frequency distribution of a parallax amount of a small region where a boundary line overlaps a small region whose parallax amount is estimated by the region parallax amount estimating unit; 3D information calculating means for calculating 3D information based on the amount of parallax estimated by the estimating means.

  In addition, the first area dividing and integrating device of the present invention includes an image input unit that inputs a plurality of image data captured at different viewpoint positions, and one reference image data among the image data input by the image input unit. An area dividing means for dividing the image into small areas, an area boundary detecting means for detecting a boundary line of each small area divided by the area dividing means, and a reference among a plurality of images input by the image input means A parallax amount estimation unit that obtains correspondence between a plurality of points on a boundary line detected by at least the region boundary line detection unit between an image and another image, and estimates a parallax amount of the plurality of points on the boundary line; A parallax amount frequency distribution creating unit that creates a frequency distribution of parallax amounts at a plurality of points on the boundary estimated by the parallax amount estimating unit for each small region, and a parallax created by the parallax amount frequency distribution creating unit A parallax amount frequency distribution analyzing unit for analyzing the frequency distribution of the amount, a region parallax amount estimating unit for estimating a parallax amount of each small region based on a result analyzed by the parallax amount frequency distribution analyzing unit, and the region parallax amount And a region integration unit that integrates the small regions according to the similarity based on the amount of parallax of each small region estimated by the estimation unit and the color information of each small region.

  The second region integration device of the present invention also includes an image input means for inputting a plurality of image data shot at different viewpoint positions, and one reference image data among the image data input by the image input means. Area dividing means for dividing into small areas, area boundary detecting means for detecting the boundary lines of the respective small areas divided by the area dividing means, and a reference image among a plurality of images input by the image input means A parallax amount estimation unit that obtains at least correspondence between a plurality of points on the boundary line detected by the region boundary line detection unit and estimates a parallax amount of the plurality of points on the boundary line between the image and another image, and the parallax A parallax amount frequency distribution creating unit that creates a frequency distribution of parallax amounts at a plurality of points on the boundary estimated by the amount estimating unit for each small region, and a parallax amount created by the parallax amount frequency distribution creating unit Parallax amount frequency distribution analyzing means for analyzing the frequency distribution, regional parallax amount estimating means for estimating the parallax amount of each small area based on the result analyzed by the parallax amount frequency distribution analyzing means, and the regional parallax amount estimating means Parallax amount frequency distribution correcting means for correcting the frequency distribution of the parallax amount in the small area where the boundary line overlaps with the small area in which the parallax amount is estimated, and the parallax amount of each small area estimated by the area parallax amount estimating means And region integration means for integrating the small regions according to the similarity based on the color information of each small region.

  In the first three-dimensional information extraction method of the present invention, one image data of a plurality of image data photographed at different viewpoint positions is used as a reference image, and the other image data is used as a search image. At least at least between the reference image and the search image, an area dividing step for dividing into small areas according to image features, an area boundary detecting step for detecting a boundary line of each small area divided by the area dividing step, A parallax amount estimation step for obtaining correspondence between a plurality of points on the boundary line of each small region detected by the region boundary line detection step, and estimating a parallax amount of the plurality of points on the boundary line of each small region; A parallax amount frequency distribution creating step for creating a frequency distribution of parallax amounts at a plurality of points on the boundary line estimated by the estimating step for each small region, and a parallax amount frequency distribution creating step A parallax amount frequency distribution analysis step for analyzing the frequency distribution of the parallax amount of each small region, and region parallax amount estimation for estimating the parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analysis step And a three-dimensional information calculation step of calculating three-dimensional information based on the parallax amount of each small region estimated in the region parallax amount estimation step.

  In the second three-dimensional information extraction method of the present invention, one image data of a plurality of image data photographed at different viewpoint positions is used as a reference image, and the other image data is used as a search image. At least at least between the reference image and the search image, an area dividing step for dividing into small areas according to image features, an area boundary detecting step for detecting a boundary line of each small area divided by the area dividing step, A parallax amount estimation step for obtaining correspondence between a plurality of points on the boundary line of each small region detected by the region boundary line detection step, and estimating a parallax amount of the plurality of points on the boundary line of each small region; A parallax amount frequency distribution creating step for creating a frequency distribution of parallax amounts at a plurality of points on the boundary line estimated by the estimating step for each small region, and a parallax amount frequency distribution creating step A parallax amount frequency distribution analysis step for analyzing the frequency distribution of the parallax amount of each small region, and region parallax amount estimation for estimating the parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analysis step A parallax amount frequency distribution correcting step for correcting a frequency distribution of a parallax amount in a small region where a boundary overlaps with a small region in which the parallax amount is estimated in the region parallax amount estimating step, and the region parallax amount estimating step And a three-dimensional information calculation step of calculating three-dimensional information based on the estimated amount of parallax.

  In the first region division integration method of the present invention, one image data of a plurality of image data photographed at different viewpoint positions is used as a reference image, and the other image data is used as a search image, and the reference image is used as an image. An area dividing step for dividing into small areas according to features, an area boundary detecting step for detecting a boundary line of each of the small areas divided by the area dividing step, and at least between the reference image and the search image A parallax estimation step for obtaining a correspondence between a plurality of points on the boundary line of each small region detected by the region boundary line detection step, and estimating a parallax amount of the plurality of points on the boundary line of each small region; A parallax amount frequency distribution creating step of creating a frequency distribution of parallax amounts of a plurality of points on the boundary estimated in the step for each small region, and the parallax amount frequency distribution creating step A parallax amount frequency distribution analyzing step for analyzing the frequency distribution of the parallax amount of each small region formed, and a region parallax amount for estimating the parallax amount of each small region based on a result analyzed by the parallax amount frequency distribution analyzing step An estimation step; and a region integration step of integrating the small regions according to the degree of similarity based on the amount of parallax of each small region estimated by the region parallax amount estimation step and the color information of each small region. .

  In the second region division integration method of the present invention, one image data among a plurality of image data photographed at different viewpoint positions is used as a reference image, and the other image data is used as a search image, and the reference image is used as an image. An area dividing step for dividing into small areas according to features, an area boundary detecting step for detecting a boundary line of each of the small areas divided by the area dividing step, and at least between the reference image and the search image A parallax estimation step for obtaining a correspondence between a plurality of points on the boundary line of each small region detected by the region boundary line detection step, and estimating a parallax amount of the plurality of points on the boundary line of each small region; A parallax amount frequency distribution creating step for creating a parallax amount frequency distribution of a plurality of points on the boundary line estimated by the step for each small region, and the parallax amount frequency distribution creating step A parallax amount frequency distribution analyzing step for analyzing the frequency distribution of the parallax amount of each small region, and region parallax amount estimation for estimating the parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analyzing step A parallax amount frequency distribution correcting step for correcting the frequency distribution of the parallax amount in a small region where the boundary line overlaps with the small region in which the parallax amount has been estimated in the region parallax amount estimating step, and the region parallax amount estimating step And a region integration step of integrating the small regions in accordance with the estimated disparity amount of each small region and the similarity based on the color information of each small region.

  Further, the first storage medium of the present invention uses one image data of a plurality of image data photographed at different viewpoint positions as a reference image, another image data as a search image, and the reference image as an image feature. And a region dividing step for dividing the region into small regions, a region boundary detecting step for detecting a boundary line of each small region divided by the region dividing step, and at least the region boundary between the reference image and the search image. By obtaining correspondence between a plurality of points on the boundary line of each small region detected by the line detection step and estimating a parallax amount of the plurality of points on the boundary line of each small region, the parallax amount estimating step A disparity amount frequency distribution creating step for creating a disparity amount frequency distribution of a plurality of points on the estimated boundary line for each small region, and each disparity amount frequency distribution creating step A parallax amount frequency distribution analyzing step for analyzing the frequency distribution of the parallax amount of the region, an area parallax amount estimating step for estimating the parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analyzing step, A computer program for executing a three-dimensional information extraction method, comprising: a three-dimensional information calculation step for calculating three-dimensional information based on the parallax amount of each small region estimated by the region parallax amount estimation step Remember as much as possible.

  Further, the second storage medium of the present invention uses one image data of a plurality of image data photographed at different viewpoint positions as a reference image, another image data as a search image, and the reference image as an image feature. And a region dividing step for dividing the region into small regions, a region boundary detecting step for detecting a boundary line of each small region divided by the region dividing step, and at least the region boundary between the reference image and the search image. By obtaining correspondence between a plurality of points on the boundary line of each small region detected by the line detection step and estimating a parallax amount of the plurality of points on the boundary line of each small region, the parallax amount estimating step A disparity amount frequency distribution creating step for creating a disparity amount frequency distribution of a plurality of points on the estimated boundary line for each small region, and each disparity amount frequency distribution creating step A parallax amount frequency distribution analyzing step for analyzing the frequency distribution of the parallax amount of the region, an area parallax amount estimating step for estimating the parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analyzing step, A parallax amount frequency distribution correcting step for correcting a frequency distribution of a parallax amount in a small region where the boundary overlaps with a small region in which the parallax amount is estimated in the region parallax amount estimating step, and the parallax estimated in the region parallax amount estimating step A computer program for executing a three-dimensional information extraction method, characterized by comprising a three-dimensional information calculation step for calculating three-dimensional information based on a quantity, is stored so as to be externally readable.

  Further, the third storage medium of the present invention uses one image data of a plurality of image data photographed at different viewpoint positions as a reference image, another image data as a search image, and uses the reference image as an image feature. And a region dividing step for dividing the region into small regions, a region boundary detecting step for detecting a boundary line of each small region divided by the region dividing step, and at least the region boundary between the reference image and the search image. By obtaining correspondence between a plurality of points on the boundary line of each small region detected by the line detection step and estimating a parallax amount of the plurality of points on the boundary line of each small region, the parallax amount estimating step A disparity amount frequency distribution creating step for creating a disparity amount frequency distribution of a plurality of points on the estimated boundary line for each small region, and each disparity amount frequency distribution creating step A parallax amount frequency distribution analyzing step for analyzing the frequency distribution of the parallax amount of the region, an area parallax amount estimating step for estimating the parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analyzing step, A region division integration method comprising: a region integration step of integrating small regions according to the degree of similarity based on the amount of parallax of each small region estimated by the region parallax amount estimation step and the color information of each small region A computer program to be executed is stored so as to be externally readable.

  Further, the fourth storage medium of the present invention uses one image data of a plurality of image data photographed at different viewpoint positions as a reference image, another image data as a search image, and the reference image as an image feature. And a region dividing step for dividing the region into small regions, a region boundary detecting step for detecting a boundary line of each small region divided by the region dividing step, and at least the region boundary between the reference image and the search image. By obtaining correspondence between a plurality of points on the boundary line of each small region detected by the line detection step and estimating a parallax amount of the plurality of points on the boundary line of each small region, the parallax amount estimating step A disparity amount frequency distribution creating step for creating a disparity amount frequency distribution of a plurality of points on the estimated boundary line for each small region, and each disparity amount frequency distribution creating step A parallax amount frequency distribution analyzing step for analyzing the frequency distribution of the parallax amount of the region, an area parallax amount estimating step for estimating the parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analyzing step, A parallax amount frequency distribution correcting step for correcting the frequency distribution of the parallax amount of a small region where the boundary line overlaps with the small region where the parallax amount is estimated by the region parallax amount estimating step, and each of the regions estimated by the region parallax amount estimating step A computer program for executing an area division integration method, comprising: an area integration step for integrating the small areas in accordance with the degree of parallax of the small areas and the similarity based on the color information of each of the small areas. To do.

  As described above, according to the present invention, the reference image is divided into small regions, the parallax of the boundary line of the small region is estimated by the region-based stereo method, and the parallax of the boundary line estimated in each region By analyzing the frequency distribution and estimating the parallax of the region, in the conventional region-based stereo method, the occurrence of miscorrespondence in the region where the change in the shade of the image is extremely small, the parallax cannot be estimated in the occlusion region, and It is possible to estimate three-dimensional information in which the problem that the boundary between the object outline and the parallax is shifted at the boundary between objects with different distances is improved. Further, by using the estimated three-dimensional information as a feature amount for integrating divided areas, it is possible to obtain an image division integration result that is closer to the original shape of the subject.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
First, a first embodiment of the present invention will be described with reference to FIGS.

  FIG. 1 is a block diagram showing the configuration of the three-dimensional information extraction apparatus in the present embodiment. In the figure, reference numeral 100 denotes a three-dimensional information extraction apparatus in the present embodiment. Reference numeral 101 denotes an image input unit that inputs a plurality of pieces of image data taken at different viewpoints. Reference numeral 102 denotes an initial region dividing unit that divides a plurality of inputted images into reference regions and divides them into small regions. An initial region integration unit 103 integrates the initial region of the reference image divided by 102 based on the color information and the region area. Reference numeral 104 denotes an area boundary detection unit that detects an area boundary formed by initial area integration. Reference numeral 105 denotes a region boundary line parallax estimation unit that searches for corresponding points of the pixels on the region boundary detected by 104 using a region-based stereo method and estimates parallax. Reference numeral 106 denotes a region boundary line parallax frequency distribution creation unit that creates a parallax frequency distribution of a region boundary line from the parallax of each pixel on the region boundary line. Reference numeral 107 denotes a region boundary line parallax frequency distribution analysis unit that analyzes the parallax frequency distribution of the region boundary line of each region and calculates the parallax value having the highest frequency in each region. Reference numeral 108 denotes an area parallax estimation unit that interpolates the parallax value calculated by 107 at the highest frequency in each area to each pixel in the area. Reference numeral 109 denotes a distance estimation unit that converts parallax values into distance information based on the principle of triangulation.

  The operation of the apparatus shown in FIG. 1 will be described. Details of the arithmetic processing of each part will be described in the description of the processing procedure. The processing of each part of the apparatus of FIG. 1 and the processing steps of FIG. 2 correspond, and the processing of 101 to 109 corresponds in order as 101 and S201, 102 and S202,..., 109 and S209. . First, a plurality of images taken at different viewpoints are input from the image input unit 101. In the initial region dividing unit 102, one of the input images is used as a reference image, and the reference image is divided into small regions based on the density of the image. Then, adjacent regions having similar color information to the small regions divided by the initial region integration unit 103 are sequentially integrated, and regions having a small area are integrated into adjacent regions having a large area. In the region boundary line detection unit 104, the boundary line of each region is detected using a pixel in which different regions are adjacent as a boundary line. The region boundary line parallax estimation unit 105 performs a corresponding point search for each pixel on the boundary line of each region by a region-based stereo method to estimate the parallax. The region boundary line parallax frequency distribution creating unit 106 creates a parallax frequency distribution from the parallax of pixels on the boundary line for each region. The parallax frequency distribution analysis unit 107 of the region boundary line calculates the parallax having the highest frequency for each region by analyzing the parallax frequency distribution of each region, and inputs the parallax to the region parallax estimation unit. The area parallax estimation unit 108 interpolates the parallax value input from 107 to all pixels in the area as the area parallax. The distance estimation unit 109 converts the parallax of each pixel into distance information to the subject based on the principle of triangulation and outputs the information.

  Next, the processing procedure of the three-dimensional information extraction method in the present embodiment will be described in detail with reference to the flowchart of FIG.

  In step S201, a reference image and a search image taken at different viewpoints are input. In S202, the reference image is divided into initial regions having a small area. Here, Watershed algorithm (L. Vincent and P. Soille, “Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations”, IEEE Transactions Pattern Analysis and Machine Intelligence, vol.13, no.6, pp.583-598, (June 1991). The Watershed algorithm is an area segmentation method that imitates the appearance of water on the topography, considering the shade of the image as topographical elevation. When water is poured into the terrain, water is filled from the low altitude basin, and when the water level rises, a watershed that separates the basin that stores the water appears. Simulating this, this is a method in which an area is expanded from a pixel having a low shading level of an image to a pixel having a low shading level of a neighboring pixel, and the image is divided into regions using a position where different regions meet as a boundary. It is characterized in that a closed boundary line can be obtained. In the present embodiment, the above-described Watershed algorithm is applied to the edge image of the brightness value to perform region division.

  Edge images are created using the Robinson operator. The Robinson operator detects edges using eight filters obtained by rotating the filters shown in FIG. 3 by 45 degrees. In this embodiment, the maximum value of the outputs of these filters is used as the edge strength.

  A region division method based on the watershed algorithm used in the present embodiment will be described. In the edge image created using the Robinson operator, first, the image is divided into a plurality of sections, and a label is given to a pixel in which the edge has a minimum value in each section. Next, among the neighboring pixels of these labels, the labels of the pixels adjacent to the pixels having relatively low edge strength in the image are propagated. However, propagation to pixels with different labels is not performed. This process is repeated to propagate each label, and labels are assigned to all pixels in the image. As a result, it is possible to perform region division having a boundary line at a portion where the edge strength is large, such as the contour of the subject. In the present embodiment, in order to equally divide the region in the image, the image is divided into sections and given initial labels.

  In step S203, the small small areas divided in step S202 are integrated based on the similarity with the adjacent areas of each area. First, the brightness, hue, and saturation of the average color are calculated from the average of the pixel values in each divided area. Then, the adjacent relationship between each small area is obtained. Next, the similarity with respect to each color component with the adjacent region is obtained by Equation (1), Equation (2), and Equation (3).

ここで

here

はそれぞれ注目する領域ｎと隣接する領域ｍの色相、明度、彩度に関する類似度である。

Are similarities regarding the hue, brightness, and saturation of the region n adjacent to the region of interest.

は注目領域ｎの平均色の色相、明度、彩度であり、

Is the hue, brightness, and saturation of the average color of the region of interest n,

は隣接領域ｍの平均色の色相、明度、彩度である。Ｈ_thは注目領域と隣接領域の色相の角度の差に対する閾値であり、色相の差が閾値より大きい場合は

Is the hue, brightness, and saturation of the average color of the adjacent region m. H _th is a threshold for the difference in hue angle between the attention area and the adjacent area, and when the difference in hue is larger than the threshold,

は１となる。Ｌ_max、Ｓ_maxは明度、彩度の最大値である。これらの類似度を統合し、隣接領域の類似度

Becomes 1. L _max and S _max are the maximum values of lightness and saturation. By integrating these similarities, the similarity of adjacent areas

を数式（４）で計算する。

Is calculated by Equation (4).

ここで、ｗ_H、ｗ_L、ｗ_Sはそれぞれ色相、明度、彩度の類似度に対する重みである。

Here, w _H , w _L , and w _S are weights for the similarity of hue, lightness, and saturation, respectively.

  The similarity between the above regions is calculated for a set of all adjacent small regions in the image, and integration is performed from the set of adjacent regions having the smallest similarity. This integration of adjacent regions is performed when the similarity between the regions is equal to or less than a threshold value. Next, the area of each region is counted, and regions having an area smaller than the block size used for the corresponding point search are integrated into adjacent regions having the smallest region similarity. By this processing, the area is larger than the block size used for the corresponding point search, and the image is divided into regions having boundary lines according to the image characteristics. This is because the ratio of the block size to the area of the region is reduced, and a large number of corresponding points of the local features of the region are obtained to obtain disparity data sufficient for the analysis of the disparity. FIG. 4 is a diagram schematically showing the processing result of the initial area integration. 4A shows the input reference image, and FIG. 4B shows the result of the initial region integration. As shown in the drawing, in the initial region integration, the subject is overdivided. Since the region integration performed here is only to be larger than the block size used for the corresponding point search, the threshold value setting of the similarity may not be strict and may be set so that the regions are not excessively integrated. In the present embodiment, the similarity between adjacent regions is set as the similarity based on the difference in lightness, hue, and saturation of the average color in the region, but other region integration methods may be used.

  In step S204, the boundary line of the area is detected. Boundary line detection is performed for each area, with pixels that are adjacent areas being different pixels as pixels constituting the boundary line.

  In step S205, for each pixel constituting the detected boundary line, a corresponding point search is performed in the search image using an area B of X × Y pixels centered on the pixel as a block. The residual similarity sum SSAD shown by Equation (5) is used for the similarity of the blocks.

ここでＴ(i,j)は基準画像中の画像ブロック内の画素の画素値を、Ｉ(i,j)は画像ブロックと比較される探索画像中の小領域内の画素の画素値を示している。Ｅ(x,y)は探索画像中を(x,y)だけ移動した位置における画素値の差の絶対値和を表している。探索範囲中で上記のＳＳＡＤの値が最小となる領域の中心を対応点として決定し、基準画像の注目点と対応点の座標のずれを視差として算出する。これを検出された境界線を構成する全画素について行う。本実施の形態では、対応点探索に用いるブロックのサイズは画像中で一定とし領域統合における領域の面積をブロックサイズを考慮して設定しているが、領域統合の結果から領域の面積に応じてブロックのサイズを適応的に変更することも可能である。また、対応点探索に用いるブロックの類似度には、残差二乗和や正規化相関係数を用いてもよい。

Here, T (i, j) indicates the pixel value of the pixel in the image block in the reference image, and I (i, j) indicates the pixel value of the pixel in the small area in the search image compared with the image block. ing. E (x, y) represents the sum of absolute values of pixel value differences at positions moved by (x, y) in the search image. The center of the region where the SSAD value is minimum in the search range is determined as a corresponding point, and a shift in coordinates between the target point of the reference image and the corresponding point is calculated as parallax. This is performed for all the pixels constituting the detected boundary line. In this embodiment, the size of the block used for the corresponding point search is constant in the image, and the area area in the area integration is set in consideration of the block size. It is also possible to adaptively change the block size. Further, a residual sum of squares or a normalized correlation coefficient may be used for the similarity of blocks used for corresponding point search.

  In step S206, the frequency distribution of the parallax values of the pixels on the boundary line is created for each region from the parallax values of the pixels on the boundary line estimated in step S205. The frequency distribution of the parallax values is created by counting the number of pixels having the parallax value for each parallax value. FIG. 5 is a graph schematically showing the frequency distribution of the parallax values of the pixels on the region boundary line. The parallax value may include an erroneous parallax value due to incorrect correspondence. In such a case, a frequency peak occurs in the original parallax value of the subject as shown in FIG. The frequency distribution is scattered.

  In step S207, the parallax value indicating the maximum frequency is extracted from the frequency distribution of the parallax values of the boundary lines for each region created in step S206. This value becomes the parallax value of the area. In the stereo method, there is a possibility that an erroneous parallax value is generated due to an incorrect correspondence. If these parallax values are used, the accuracy of three-dimensional information extraction is deteriorated. Therefore, in this embodiment, the parallax value indicating the maximum frequency among the parallax values of the pixels on the region boundary line is set as the parallax value of the region, thereby removing the influence of the parallax value due to incorrect correspondence and the accuracy of the three-dimensional information extraction. It is possible to improve the problem of deterioration.

  In step S208, the parallax value of each area calculated in step S207 is interpolated to all pixels in the area. Thereby, the parallax interpolation is performed on all the pixels in the image.

  In step S209, the distance from the imaging device to the subject is calculated based on the principle of triangulation for the parallax of each pixel.

  As described above in detail, an area with extremely few image features can be obtained by extracting a boundary line having an image characteristic by area division reflecting the contour of the subject and using the area-based stereo method only for the boundary line. Because it is possible to obtain parallax data for statistical analysis to suppress the occurrence of miscorrespondence and eliminate the effects of miscorrespondence according to the image, easy processing of disparity with few errors in all images Can be obtained. Since only the reference image needs to be divided into regions, there is no need to extract the features equally between the reference image and the search image as in the conventional feature-based stereo method, and it is possible to search for corresponding points with high reliability through simple processing. It is. Since the highest value of the frequency distribution of the parallax of the area is used as the parallax value of the area, it is possible to reduce the influence of erroneous correspondence in the occlusion area and the contour portion of the object. As a result, the parallax variation is small, and the three-dimensional information can be extracted with high accuracy even in the contour portion of the subject. In addition, processing according to the image is possible, and setting of each processing is easy and has a wide allowable range, so that robust processing can be realized for complex images and natural scenes.

(Second embodiment)
A second embodiment of the present invention will be described with reference to FIGS.

  FIG. 6 is a block diagram showing a configuration of the three-dimensional information extraction apparatus in the present embodiment. In the figure, reference numeral 600 denotes a three-dimensional information extraction apparatus in the present embodiment. Reference numeral 601 denotes an image input unit for inputting a plurality of pieces of image data taken at different viewpoints. Reference numeral 602 denotes an initial region dividing unit that divides one of a plurality of input images as a reference image and divides it into small regions. Reference numeral 603 denotes an initial region integration unit that integrates the initial region of the reference image divided by 602 based on the color information and the region area. An area boundary detection unit 604 detects an area boundary formed by initial area integration. A region boundary line parallax estimation unit 605 searches for corresponding points using a region-based stereo method and estimates parallax for pixels on the region boundary line detected by 604. A region boundary line parallax frequency distribution creating unit 606 creates a parallax frequency distribution of the region boundary line from the parallax of each pixel on the region boundary line. Reference numeral 607 denotes a parallax frequency distribution analysis unit for the region boundary line that analyzes the parallax frequency distribution of the region boundary line of each region and calculates the parallax average value, variance, and parallax value having the highest frequency in each region. Reference numeral 608 denotes an area parallax estimation unit that determines the parallax estimation area by determining the average value and variance of the parallax calculated in 607 and estimates the parallax of the area using the maximum frequency parallax. Reference numeral 609 denotes a region boundary line parallax frequency distribution correction unit that corrects the parallax frequency distribution of a region adjacent to the region where the parallax estimation is performed. Reference numeral 610 denotes an area parallax interpolation unit that interpolates the parallax of an area where parallax estimation cannot be performed from the parallax of an adjacent area. Reference numeral 611 denotes a distance estimation unit that converts parallax values into distance information based on the principle of triangulation.

  The operation of the apparatus shown in FIG. 6 will be described. Details of the arithmetic processing of each part will be described in the description of the processing procedure. The processing of each part of the apparatus of FIG. 6 and the processing steps of FIG. 7 correspond to each other, and the processing in 601 to 611 sequentially corresponds to 601 and S701, 602 and S702,... 611 and S711. . Reference numerals 601 to 605 denote the same operations as 101 to 105 in the first embodiment, and a description thereof will be omitted. In the region boundary line parallax frequency distribution creating unit 606, a parallax frequency distribution is created from the parallax of each pixel on the boundary line for each region. At this time, the weight of the contribution to the frequency distribution of the parallax of each pixel is changed according to the edge strength of each pixel. The region boundary parallax frequency distribution analysis unit 607 analyzes the parallax frequency distribution of each region, calculates the parallax average value, variance, and maximum frequency parallax value of each region, and sends them to the region parallax estimation unit. Entered. The area parallax estimation unit 608 determines the average value and variance of the parallax of each area input from 607, and calculates the parallax value of the maximum frequency input from 607 in order from the area where the average value is high and the variance is small. Are interpolated to all pixels in the region as the parallax. The region boundary line parallax frequency distribution correction unit 609 removes the parallax of the overlapping portion of the boundary line from the parallax frequency distribution of the region where the boundary overlaps with the estimated parallax region. The parallax estimation of the area and the correction of the frequency distribution are repeatedly performed, and the parallax of the area is estimated in order. In the area parallax interpolation unit 610, the parallax of the area where the parallax cannot be estimated has already been estimated, and is interpolated using the parallax of the adjacent area having a similar average color. The distance estimation unit 611 converts the parallax of each pixel into distance information to the subject based on the principle of triangulation and outputs the information.

  Next, the processing procedure of the three-dimensional information extraction method in the present embodiment will be described in detail with reference to the flowchart of FIG.

  The processes from step S701 to S705 are the same as steps S201 to S205 in the first embodiment, and the description thereof is omitted here. In step S706, a parallax frequency distribution is created from the parallax of each pixel on the boundary line for each region estimated in step S705. Here, unlike the first embodiment, the contribution of each pixel to the frequency distribution of parallax is changed according to the edge strength of the pixel on the boundary line as follows.

C _n (i, j) = D (i, j) / D _max (6)
Here, C _n (i, j) is a contribution to the frequency distribution of the parallax of the pixels on the boundary line of the coordinate (i, j) of the region n, and D (i, j) is the coordinate (i, j) The edge strength, D _max is the maximum value of the edge strength. The stronger the edge strength, the greater the contribution to the frequency distribution of the parallax, and the greater the contribution from the pixels where the image features are strong and the region-based stereo method is more likely to estimate the parallax more reliable. It is possible to estimate the parallax of the region with high accuracy. The change in the contribution to the frequency distribution of the parallax is not limited to the equation (6), and the contribution to the frequency distribution of the parallax may be increased as the edge strength increases.

  In step S707, the frequency distribution of the parallax on the boundary line is analyzed for each region. In step S707-1, the parallax frequency distribution on the boundary line is analyzed for each region to calculate the parallax average value and variance. In step S707-2, the parallax value having the maximum frequency is obtained from the parallax frequency distribution. Although the parallax value having the maximum frequency is obtained in the present embodiment, the parallax value having the maximum frequency within a predetermined range centered on the parallax value may be used.

  In step S708, the parallax of each area is estimated. In step S708-1, an area for which parallax estimation is performed is determined from the average value and variance of parallax on the boundary line of each area. Here, a region where parallax estimation is performed in the order of higher average value and smaller variance is determined. FIG. 8 is a graph schematically showing the frequency distribution of parallax of pixels on the region boundary line. When the boundary line of an area is formed by the contours of objects with different distances in the image, the parallax frequency distribution on the boundary line is multimodal as shown in FIG. 8A, and the parallax variance is high. Moreover, it can be said that the fact that the variance is small has little influence of erroneous parallax such as erroneous correspondence. For this reason, the smaller the variance, the higher the possibility that the region has uniform parallax and highly reliable parallax estimation is performed, and the parallax is preferentially determined from such a region. The boundary line formed by the contours of objects with different distances in the image is generated by being shielded by objects with close distances, and part of the boundary lines reflect the contours of objects with close distances. Parallax is the parallax of an object with a short distance. Therefore, by estimating the area from objects that are close to each other and removing the parallax of the overlapping area with the boundary line of the area from the frequency distribution of the parallax of the adjacent area where the area and the boundary line overlap, the distance is close The parallax of the region can be estimated without being affected by the parallax of the object. For this reason, the parallax of the area is preferentially given to the area having a high average parallax value. Based on the parallax characteristics of the boundary line of such an area, parallax can be estimated with higher accuracy by estimating the parallax in order from an area having a high average value and a small variance. In the present embodiment, it is determined whether or not the region has a close distance and a uniform parallax based on the average value and the variance. However, other evaluation measures may be used. In step S708-2, for an area determined to be subjected to parallax estimation from the average and variance of parallax on the area boundary line, parallax interpolation is performed on all pixels in the area with the parallax value of the maximum frequency of the area. In step S708-3, it is determined whether or not all region boundary lines are used for parallax estimation. If all region boundary lines are used for parallax estimation, the process proceeds to step S710. If there is an area boundary line that is not used for parallax estimation, the process proceeds to step S709.

  In step S709, the disparity of the overlapping portion is removed from the disparity frequency distribution of the adjacent region where the boundary line overlaps with the region where the disparity estimation has been performed. FIG. 8B is a graph schematically showing the result of correcting the frequency distribution of FIG. The parallax of the boundary line portion formed by the object having a small distance from the imaging device is removed, and the frequency distribution having a unimodal peak in the original parallax value of the region is corrected. By estimating the parallax from an area with a small distance from the imaging device and removing the parallax of the estimated boundary line from the parallax of the boundary line of the adjacent area, the contours of multiple objects with different distances The parallax of the region can be accurately estimated without using the parallax of the contour of the object with a small distance with respect to the region where the boundary line is formed. After this process is completed, the process returns to step S707, the processes of S707 to S708 are repeated, and the parallax of the region is sequentially estimated.

  In step S710, it is determined whether or not the parallax of all the regions has been estimated in step S710-1, and if the parallax of all the regions has been estimated, the process proceeds to step S711. If there is an area where the parallax has not been estimated, the process proceeds to step S710-2, where the same degree of similarity as in the initial area integration is used and the parallax of the adjacent area having the smallest degree of similarity between adjacent areas is used as the area parallax. The parallax is interpolated to the pixels.

  In step S711, the distance from the imaging device to the subject is calculated based on the principle of triangulation as in the first embodiment.

  As described in detail above, the order of parallax estimation is determined from the parallax average value and variance calculated by analyzing the parallax frequency distribution of the area boundary, and the area where the parallax estimation is performed overlaps with the boundary This is caused by removing the parallax of the overlapping part from the frequency distribution of the parallax in the adjacent area, and by blocking the near-side subject in the region where the boundary line is formed by a plurality of objects at different distances such as the contour portion of the subject. The parallax of the boundary line can be removed, and accurate three-dimensional information can be extracted without any deviation between the contour of the subject and the parallax boundary. Also, when creating the disparity frequency distribution, the contribution of the disparity of each pixel on the boundary line is changed according to the edge strength, the contribution of the disparity of the pixel that has a strong feature and reliable disparity estimation is increased, It is possible to further improve the accuracy of the parallax estimation of the region by reducing the contribution of the parallax of the pixel that is less likely to cause an erroneous response.

(Third embodiment)
First, a third embodiment of the present invention will be described with reference to FIGS.

  FIG. 9 is a block diagram showing a configuration of the area division and integration device according to the present embodiment. In the figure, reference numeral 900 denotes an area division integration apparatus according to the present embodiment. Reference numeral 901 denotes an image input unit that inputs a plurality of pieces of image data taken at different viewpoints. Reference numeral 902 denotes an initial region dividing unit that divides one of a plurality of input images as a reference image and divides it into small regions. Reference numeral 903 denotes an initial region integration unit that integrates the initial region of the reference image divided by 902 based on the color information and the region area. An area boundary detection unit 904 detects an area boundary formed by initial area integration. Reference numeral 905 denotes a region boundary line parallax estimation unit that searches for corresponding points of the pixels on the region boundary line detected by 904 using a region-based stereo method and estimates parallax. A region boundary line parallax frequency distribution creating unit 906 creates a parallax frequency distribution of the region boundary line from the parallax of each pixel on the region boundary line. Reference numeral 907 denotes a region boundary line parallax frequency distribution analysis unit that analyzes the parallax frequency distribution of the region boundary line of each region and calculates the parallax value having the highest frequency in each region. Reference numeral 908 denotes an area parallax estimation unit that interpolates the parallax value calculated by 907 with the highest frequency in each area to each pixel in the area. Reference numeral 909 denotes an area integration unit that integrates areas based on color information and parallax information.

  The operation of the apparatus shown in FIG. 9 will be described. Details of the arithmetic processing of each part will be described in the description of the processing procedure. The processing of each part of the apparatus of FIG. 9 and the processing steps of FIG. 10 correspond to each other, and the processing in 901 to 909 sequentially corresponds to 901 and S1001, 902 and S1002,... 909 and S1009. . Reference numerals 901 to 908 denote the same operations as those in the first embodiment, and a description thereof will be omitted. The region integration unit 909 performs region integration based on the similarity of the average color of each region integrated by the initial region integration unit 903 and the similarity of the parallax values of each region estimated up to 908, and results of region division integration Is output.

  Next, the processing procedure of the region division integration method according to the present embodiment will be described in detail with reference to the flowchart of FIG.

  The processing procedure from step S1001 to S1008 is the same as that of the first embodiment. In step S1009, region integration is performed based on the average color similarity and the disparity value similarity of each region. Similar to the first embodiment, the average color similarity is represented by the equation (4).

を用いる。視差値の類似度は数式（７）を用いて算出する。

Is used. The similarity between the parallax values is calculated using Equation (7).

ここで、

here,

は注目する領域ｎと隣接する領域ｍの視差値の類似度である。

Is the degree of similarity of the parallax values between the region n of interest and the adjacent region m.

はそれぞれ注目領域の視差値、隣接領域の視差値である。Ｐ_maxは画像中の視差値の最大値である。これらの各類似度から隣接領域間の類似度ΔＳim_mnを数式（８）で算出する。

Are the parallax value of the attention area and the parallax value of the adjacent area, respectively. P _max is the maximum parallax value in the image. From these similarities, the similarity ΔSim _mn between adjacent regions is calculated by Equation (8).

ここでｗ_V、ｗ_Pはそれぞれ平均色および視差の類似度に対する重みである。画像中の隣接するすべての領域の組について上記の領域間の類似度を算出し、類似度が最も小さい隣接領域の組から統合を行っていく。この隣接領域の統合は領域間の類似度が閾値以下の場合に行われる。本実施の形態では、平均色の類似度および視差の類似度を用いたが、色に関する他の類似度や、視差を変換することにより得られる距離情報などを用いてもよい。

Here, w _V and w _P are weights for the similarity between the average color and the parallax, respectively. The similarity between the above regions is calculated for a set of all adjacent regions in the image, and integration is performed from the set of adjacent regions having the smallest similarity. This integration of adjacent regions is performed when the similarity between the regions is equal to or less than a threshold value. In this embodiment, the similarity of the average color and the similarity of the parallax are used, but other similarities regarding the color, distance information obtained by converting the parallax, and the like may be used.

  As described above in detail, by using the parallax information that does not cause a deviation between the boundary between the object contour and the parallax for the region integration, the region integration can be easily performed even for subjects having many different colors, and the subject contour As a result, it is possible to obtain a region division integration result corresponding to an image in which is accurately reflected.

  In the present embodiment, the region division is performed by the Watershed algorithm. However, for example, a region growing method or the like may be used.

  In this embodiment, the parallax is converted into a distance from the imaging device based on the principle of triangulation and output. However, as a form of output of the three-dimensional information, the parallax value is output without conversion. May be. Further, the present invention includes the case of outputting in another form in which the parallax value or the distance from the imaging device is converted.

  Further, the present invention supplies a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus stores the storage medium. It can also be applied to reading and executing the program code stored in.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

  As a storage medium for supplying the program code, for example, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like is used. be able to.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) running on the computer based on the instruction of the program code. Needless to say, the present invention includes a case where the functions of the embodiment described above are realized by performing part or all of the actual processing.

  Further, after the program code read from the storage medium is written to a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

It is a block diagram which shows the structure of the three-dimensional information extraction apparatus in the 1st Embodiment of this invention. It is a flowchart which shows the process sequence of the three-dimensional information extraction method in the 1st Embodiment of this invention. It is a figure which shows the Robinson operator used when producing the edge image in the 1st Embodiment of this invention. FIG. 6 is a diagram schematically showing an initial region integration result in the first exemplary embodiment of the present invention. It is a graph which shows typically the creation result of the parallax frequency distribution of the region boundary in the 1st embodiment of the present invention. It is a block diagram which shows the structure of the three-dimensional information extraction apparatus in the 2nd Embodiment of this invention. It is a flowchart which shows the process sequence of the three-dimensional information extraction method in the 2nd Embodiment of this invention. It is a graph which shows typically the creation result of the parallax frequency distribution of the region boundary in the 2nd embodiment of the present invention. It is a block diagram which shows the structure of the area division integration apparatus in the 3rd Embodiment of this invention. It is a flowchart which shows the process sequence of the area division | segmentation integration method in the 3rd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 3D information extraction apparatus 101 Image input part 102 Initial area division part 103 Initial area integration part 104 Area boundary line detection part 105 Area boundary line parallax estimation part 106 Area boundary line parallax frequency distribution creation part 107 Area boundary line parallax Frequency distribution analysis unit 108 Region parallax estimation unit 109 Distance estimation unit 600 Three-dimensional information extraction device 601 Image input unit 602 Initial region division unit 603 Initial region integration unit 604 Region boundary line detection unit 605 Region boundary line parallax estimation unit 606 region Boundary line parallax frequency distribution creation unit 607 Region boundary line parallax frequency distribution analysis unit 608 Region parallax estimation unit 609 Region boundary line parallax frequency distribution correction unit 610 Region parallax interpolation unit 611 Distance estimation unit 900 Region division integration device 901 Image input Part 902 initial area dividing part 903 initial area integrating part 904 area boundary line inspection Part 905 regions border parallax frequency distribution analysis unit 908 regional disparity estimator 909 region integrating part of the parallax frequency distribution creation unit 907 region boundary line of the parallax estimation unit 906 regions border

Claims (30)

Image input means for inputting a plurality of image data shot at different viewpoint positions;
Area dividing means for dividing one reference image data of the image data input by the image input means into small areas;
Area boundary detection means for detecting a boundary line of each small area divided by the area dividing means;
Between at least a reference image of the plurality of images input by the image input means and another image, and at least correspondence of a plurality of points on the boundary line of each small region detected by the region boundary line detection unit; Parallax amount estimation means for estimating the parallax amount of a plurality of points on the boundary line of the small region;
A parallax amount frequency distribution creating unit that creates a frequency distribution of the parallax amount of a plurality of points on the boundary estimated by the parallax amount estimating unit for each small region;
A parallax amount frequency distribution analyzing unit that analyzes the frequency distribution of the parallax amount of each small region created by the parallax amount frequency distribution creating unit;
Area parallax amount estimating means for estimating the amount of parallax of each small area based on the result analyzed by the parallax amount frequency distribution analyzing means;
Three-dimensional information calculation means for calculating three-dimensional information based on the parallax amount of each small area estimated by the area parallax amount estimation means;
A three-dimensional information extraction apparatus comprising: The parallax amount frequency distribution analyzing means calculates the parallax amount of the maximum frequency for each small region from the frequency distribution of the parallax amount of each small region,
2. The three-dimensional information extraction device according to claim 1, wherein the region parallax amount estimation unit estimates the parallax amount of each small region using the parallax amount of the maximum frequency of each small region as the parallax amount of the small region. . Image input means for inputting a plurality of image data shot at different viewpoint positions;
Area dividing means for dividing one reference image data of the image data input by the image input means into small areas;
Area boundary detection means for detecting a boundary line of each small area divided by the area dividing means;
Between at least a reference image of the plurality of images input by the image input means and another image, and at least correspondence of a plurality of points on the boundary line of each small region detected by the region boundary line detection unit; Parallax amount estimation means for estimating the parallax amount of a plurality of points on the boundary line of the small region;
A parallax amount frequency distribution creating unit that creates a frequency distribution of the parallax amount of a plurality of points on a boundary line estimated by the parallax amount estimating unit for each small region;
A parallax amount frequency distribution analyzing unit that analyzes the frequency distribution of the parallax amount of each small region created by the parallax amount frequency distribution creating unit;
Area parallax amount estimating means for estimating the amount of parallax of each small area based on the result analyzed by the parallax amount frequency distribution analyzing means;
A parallax amount frequency distribution correcting unit that corrects the frequency distribution of the parallax amount in a small region where the boundary line overlaps with the small region in which the parallax amount is estimated by the region parallax amount estimating unit;
Three-dimensional information calculation means for calculating three-dimensional information based on the amount of parallax estimated by the area parallax amount estimation means;
A three-dimensional information extraction apparatus comprising: The parallax amount frequency distribution analysis means calculates an average value of the parallax amount, a variance and a parallax amount of the maximum frequency for each small region from the frequency distribution of the parallax amount of each small region,
The region parallax amount estimation means is configured so that the average value of the parallax amount is relatively high in the small regions in the image, and the variance of the parallax amount is in order from the relatively small regions in the small regions in the image. , Estimating the amount of parallax of each small region by using the parallax amount of the maximum frequency of each small region as the amount of parallax of the small region,
The parallax amount frequency distribution correcting unit removes the parallax amount of the portion where the boundary line overlaps from the parallax amount frequency distribution of the small region where the boundary line overlaps with the small region where the parallax amount is estimated by the region parallax amount estimating unit. The three-dimensional information extraction apparatus according to claim 3.   The parallax amount frequency distribution creating means creates the frequency distribution by changing the contribution of the parallax amount of each point to the frequency distribution according to the image feature amount of each point on the boundary line of each small region. The three-dimensional information extraction device according to any one of claims 1 to 4. Image input means for inputting a plurality of image data shot at different viewpoint positions;
Area dividing means for dividing one reference image data of the image data input by the image input means into small areas;
Area boundary detection means for detecting a boundary line of each small area divided by the area dividing means;
A correspondence between at least a plurality of points on the boundary line detected by the region boundary line detection unit is obtained between a reference image of the plurality of images input by the image input unit and another image, and a plurality of points on the boundary line are obtained. Parallax amount estimation means for estimating the parallax amount of the point;
A parallax amount frequency distribution creating unit that creates a frequency distribution of the parallax amount of a plurality of points on a boundary line estimated by the parallax amount estimating unit for each small region;
A parallax amount frequency distribution analyzing means for analyzing the frequency distribution of the parallax amount created by the parallax amount frequency distribution creating means;
Area parallax amount estimating means for estimating the amount of parallax of each small area based on the result analyzed by the parallax amount frequency distribution analyzing means;
Area integration means for integrating the small areas according to the similarity based on the parallax amount of each small area estimated by the area parallax amount estimation means and the color information of each small area;
An area division and integration device comprising: The parallax amount frequency distribution analyzing means calculates the parallax amount of the maximum frequency for each small region from the frequency distribution of the parallax amount of each small region,
7. The area division integration apparatus according to claim 6, wherein the area parallax amount estimation unit estimates the parallax amount of each small area using the parallax amount of the maximum frequency of each small area as the parallax amount of the small area. Image input means for inputting a plurality of image data shot at different viewpoint positions;
Area dividing means for dividing one reference image data of the image data input by the image input means into small areas;
Area boundary detection means for detecting a boundary line of each small area divided by the area dividing means;
A correspondence between at least a plurality of points on the boundary line detected by the region boundary line detection unit is obtained between a reference image of the plurality of images input by the image input unit and another image, and a plurality of points on the boundary line are obtained. Parallax amount estimation means for estimating the parallax amount of the point;
A parallax amount frequency distribution creating unit that creates a frequency distribution of the parallax amount of a plurality of points on a boundary line estimated by the parallax amount estimating unit for each small region;
A parallax amount frequency distribution analyzing means for analyzing the frequency distribution of the parallax amount created by the parallax amount frequency distribution creating means;
Area parallax amount estimating means for estimating the amount of parallax of each small area based on the result analyzed by the parallax amount frequency distribution analyzing means;
A parallax amount frequency distribution correcting unit that corrects the frequency distribution of the parallax amount of the small region where the boundary line overlaps with the small region of which the parallax amount is estimated by the region parallax amount estimating unit;
Area integration means for integrating the small areas according to the similarity based on the parallax amount of each small area estimated by the area parallax amount estimation means and the color information of each small area;
An area division and integration device comprising: The parallax amount frequency distribution analysis means calculates an average value of the parallax amount, a variance and a parallax amount of the maximum frequency for each small region from the frequency distribution of the parallax amount of each small region,
The region parallax amount estimation means is configured so that the average value of the parallax amount is relatively high in the small regions in the image, and the variance of the parallax amount is in order from the relatively small regions in the small regions in the image. , Estimating the amount of parallax of each small region by using the parallax amount of the maximum frequency of each small region as the amount of parallax of the small region,
The parallax amount frequency distribution correcting unit removes the parallax amount of the portion where the boundary line overlaps from the parallax amount frequency distribution of the small region where the boundary line overlaps with the small region where the parallax amount is estimated by the region parallax amount estimating unit. The area division integration apparatus according to claim 8.   The parallax amount frequency distribution creating means creates the frequency distribution by changing the contribution of the parallax amount of each point to the frequency distribution according to the image feature amount of each point on the boundary line of each small region. The area division integration apparatus according to any one of claims 5 to 9. Of the plurality of image data taken at different viewpoint positions, one image data is a reference image, and the other image data is a search image.
An area dividing step of dividing the reference image into small areas according to image characteristics;
A region boundary detection step for detecting a boundary line of each small region divided by the region division step;
A correspondence between a plurality of points on the boundary line of each small region detected by at least the region boundary line detection step between the reference image and the search image is obtained, and a parallax amount of the plurality of points on the boundary line of each small region is obtained. A parallax amount estimation step to be estimated;
A parallax amount frequency distribution creating step for creating a frequency distribution of parallax amounts at a plurality of points on the boundary estimated by the parallax amount estimating step for each small region;
A parallax amount frequency distribution analyzing step of analyzing the frequency distribution of the parallax amount of each small region created by the parallax amount frequency distribution creating step;
A region parallax amount estimation step for estimating a parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analysis step;
A three-dimensional information calculation step for calculating three-dimensional information based on the parallax amount of each small region estimated by the region parallax amount estimation step;
A three-dimensional information extraction method characterized by comprising: The parallax amount frequency distribution analysis step calculates the parallax amount of the maximum frequency for each small region from the frequency distribution of the parallax amount of each small region,
12. The three-dimensional information extraction method according to claim 11, wherein the region parallax amount estimating step estimates the parallax amount of each small region using the parallax amount of the maximum frequency of each small region as the parallax amount of the small region. . Of the plurality of image data taken at different viewpoint positions, one image data is a reference image, and the other image data is a search image.
An area dividing step of dividing the reference image into small areas according to image characteristics;
A region boundary detection step for detecting a boundary line of each small region divided by the region division step;
A correspondence between a plurality of points on the boundary line of each small region detected by at least the region boundary line detection step between the reference image and the search image is obtained, and a parallax amount of the plurality of points on the boundary line of each small region is obtained. A parallax amount estimation step to be estimated;
A parallax amount frequency distribution creating step for creating a frequency distribution of parallax amounts at a plurality of points on the boundary estimated by the parallax amount estimating step for each small region;
A parallax amount frequency distribution analyzing step of analyzing the frequency distribution of the parallax amount of each small region created by the parallax amount frequency distribution creating step;
A region parallax amount estimation step for estimating a parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analysis step;
A parallax amount frequency distribution correction step for correcting the frequency distribution of the parallax amount of a small region where the boundary line overlaps with the small region whose parallax amount is estimated by the region parallax amount estimation step;
A three-dimensional information calculation step for calculating three-dimensional information based on the amount of parallax estimated by the region parallax amount estimation step;
A three-dimensional information extraction method characterized by comprising: The parallax amount frequency distribution analysis step calculates a parallax amount average value, variance, and maximum frequency parallax amount for each small region from the frequency distribution of the parallax amount of each small region;
The region parallax amount estimation step is performed in order from a small region in which the average value of the parallax amount is relatively high in the small region in the image and the variance of the parallax amount is relatively small in the small region in the image. , Estimating the amount of parallax of each small region by using the parallax amount of the maximum frequency of each small region as the amount of parallax of the small region,
The parallax amount frequency distribution correcting step removes the parallax amount of the portion where the boundary line overlaps from the parallax amount frequency distribution of the small region where the boundary line overlaps with the small region where the parallax amount is estimated by the region parallax amount estimating step. The three-dimensional information extraction method according to claim 13.   The parallax amount frequency distribution creating step creates the frequency distribution by changing the contribution of the parallax amount of each point to the frequency distribution according to the image feature amount of each point on the boundary line of each small region. The three-dimensional information extraction method according to claim 11. Of the plurality of image data taken at different viewpoint positions, one image data is a reference image, and the other image data is a search image.
An area dividing step of dividing the reference image into small areas according to image characteristics;
A region boundary detection step for detecting a boundary line of each small region divided by the region division step;
A correspondence between a plurality of points on the boundary line of each small region detected by at least the region boundary line detection step between the reference image and the search image is obtained, and a parallax amount of the plurality of points on the boundary line of each small region is obtained. A parallax amount estimation step to be estimated;
A parallax amount frequency distribution creating step for creating a frequency distribution of parallax amounts at a plurality of points on the boundary estimated by the parallax amount estimating step for each small region;
A parallax amount frequency distribution analyzing step of analyzing the frequency distribution of the parallax amount of each small region created by the parallax amount frequency distribution creating step;
A region parallax amount estimation step for estimating a parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analysis step;
A region integration step of integrating the small regions according to the degree of similarity based on the parallax amount of each small region estimated by the region parallax amount estimation step and the color information of each small region;
An area dividing and integrating method characterized by comprising: The parallax amount frequency distribution analysis step calculates the parallax amount of the maximum frequency for each small region from the frequency distribution of the parallax amount of each small region,
The region division integration method according to claim 16, wherein the region parallax amount estimating step estimates the parallax amount of each small region by using the parallax amount of the maximum frequency of each small region as the parallax amount of the small region. Of the plurality of image data taken at different viewpoint positions, one image data is a reference image, and the other image data is a search image.
An area dividing step of dividing the reference image into small areas according to image characteristics;
A region boundary detection step for detecting a boundary line of each small region divided by the region division step;
A correspondence between a plurality of points on the boundary line of each small region detected by at least the region boundary line detection step between the reference image and the search image is obtained, and a parallax amount of the plurality of points on the boundary line of each small region is obtained. A parallax amount estimation step to be estimated;
A parallax amount frequency distribution creating step for creating a frequency distribution of parallax amounts at a plurality of points on the boundary estimated by the parallax amount estimating step for each small region;
A parallax amount frequency distribution analyzing step of analyzing the frequency distribution of the parallax amount of each small region created by the parallax amount frequency distribution creating step;
A region parallax amount estimation step for estimating a parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analysis step;
A parallax amount frequency distribution correction step for correcting the frequency distribution of the parallax amount of a small region where the boundary line overlaps with the small region whose parallax amount is estimated by the region parallax amount estimation step;
A region integration step of integrating the small regions according to the degree of similarity based on the parallax amount of each small region estimated by the region parallax amount estimation step and the color information of each small region;
An area dividing and integrating method characterized by comprising: The parallax amount frequency distribution analysis step calculates a parallax amount average value, variance, and maximum frequency parallax amount for each small region from the frequency distribution of the parallax amount of each small region;
The region parallax amount estimation step is performed in order from a small region in which the average value of the parallax amount is relatively high in the small region in the image and the variance of the parallax amount is relatively small in the small region in the image. , Estimating the amount of parallax of each small region by using the parallax amount of the maximum frequency of each small region as the amount of parallax of the small region,
The parallax amount frequency distribution correcting step removes the parallax amount of the portion where the boundary line overlaps from the parallax amount frequency distribution of the small region where the boundary line overlaps with the small region where the parallax amount is estimated by the region parallax amount estimating step. The area division integration method according to claim 18.   The parallax amount frequency distribution creating step creates the frequency distribution by changing the contribution of the parallax amount of each point to the frequency distribution according to the image feature amount of each point on the boundary line of each small region. The area division integration method according to any one of claims 16 to 19. Of the plurality of image data taken at different viewpoint positions, one image data is a reference image, and the other image data is a search image.
An area dividing step of dividing the reference image into small areas according to image characteristics;
A region boundary detection step for detecting a boundary line of each small region divided by the region division step;
A correspondence between a plurality of points on the boundary line of each small region detected by at least the region boundary line detection step between the reference image and the search image is obtained, and a parallax amount of the plurality of points on the boundary line of each small region is obtained. A parallax amount estimation step to be estimated;
A parallax amount frequency distribution creating step for creating a frequency distribution of parallax amounts at a plurality of points on the boundary estimated by the parallax amount estimating step for each small region;
A parallax amount frequency distribution analyzing step of analyzing the frequency distribution of the parallax amount of each small region created by the parallax amount frequency distribution creating step;
A region parallax amount estimation step for estimating a parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analysis step;
A three-dimensional information calculation step for calculating three-dimensional information based on the parallax amount of each small region estimated by the region parallax amount estimation step;
A storage medium for storing a computer program for executing a three-dimensional information extraction method characterized by comprising: The parallax amount frequency distribution analysis step calculates the parallax amount of the maximum frequency for each small region from the frequency distribution of the parallax amount of each small region,
The storage medium according to claim 21, wherein the regional parallax amount estimation step estimates the parallax amount of each small region using the parallax amount of the maximum frequency of each small region as the parallax amount of the small region. Of the plurality of image data taken at different viewpoint positions, one image data is a reference image, and the other image data is a search image.
An area dividing step of dividing the reference image into small areas according to image characteristics;
A region boundary detection step for detecting a boundary line of each small region divided by the region division step;
A correspondence between a plurality of points on the boundary line of each small region detected by at least the region boundary line detection step between the reference image and the search image is obtained, and a parallax amount of the plurality of points on the boundary line of each small region is obtained. A parallax amount estimation step to be estimated;
A parallax amount frequency distribution creating step for creating a frequency distribution of parallax amounts at a plurality of points on the boundary estimated by the parallax amount estimating step for each small region;
A parallax amount frequency distribution analyzing step of analyzing the frequency distribution of the parallax amount of each small region created by the parallax amount frequency distribution creating step;
A region parallax amount estimation step for estimating a parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analysis step;
A parallax amount frequency distribution correction step for correcting the frequency distribution of the parallax amount of a small region where the boundary line overlaps with the small region whose parallax amount is estimated by the region parallax amount estimation step;
A three-dimensional information calculation step for calculating three-dimensional information based on the amount of parallax estimated by the region parallax amount estimation step;
A storage medium for storing a computer program for executing a three-dimensional information extraction method characterized by comprising: The parallax amount frequency distribution analysis step calculates a parallax amount average value, variance, and maximum frequency parallax amount for each small region from the frequency distribution of the parallax amount of each small region;
The region parallax amount estimation step is performed in order from a small region in which the average value of the parallax amount is relatively high in the small region in the image and the variance of the parallax amount is relatively small in the small region in the image. , Estimating the amount of parallax of each small region by using the parallax amount of the maximum frequency of each small region as the amount of parallax of the small region,
The parallax amount frequency distribution correcting step removes the parallax amount of the portion where the boundary line overlaps from the parallax amount frequency distribution of the small region where the boundary line overlaps with the small region where the parallax amount is estimated by the region parallax amount estimating step. 24. The storage medium according to claim 23.   The parallax amount frequency distribution creating step creates the frequency distribution by changing the contribution of the parallax amount of each point to the frequency distribution according to the image feature amount of each point on the boundary line of each small region. The storage medium according to any one of claims 21 to 24. Of the plurality of image data taken at different viewpoint positions, one image data is a reference image, and the other image data is a search image.
An area dividing step of dividing the reference image into small areas according to image characteristics;
A region boundary detection step for detecting a boundary line of each small region divided by the region division step;
A correspondence between a plurality of points on the boundary line of each small region detected by at least the region boundary line detection step between the reference image and the search image is obtained, and a parallax amount of the plurality of points on the boundary line of each small region is obtained. A parallax amount estimation step to be estimated;
A parallax amount frequency distribution creating step for creating a frequency distribution of parallax amounts at a plurality of points on the boundary estimated by the parallax amount estimating step for each small region;
A parallax amount frequency distribution analyzing step of analyzing the frequency distribution of the parallax amount of each small region created by the parallax amount frequency distribution creating step;
A region parallax amount estimation step for estimating a parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analysis step;
A region integration step of integrating the small regions according to the degree of similarity based on the parallax amount of each small region estimated by the region parallax amount estimation step and the color information of each small region;
A storage medium for storing a computer program for executing an area division integration method characterized by comprising: The parallax amount frequency distribution analysis step calculates the parallax amount of the maximum frequency for each small region from the frequency distribution of the parallax amount of each small region,
27. The storage medium according to claim 26, wherein the regional parallax amount estimating step estimates the parallax amount of each small region using the parallax amount of the maximum frequency of each small region as the parallax amount of the small region. Of the plurality of image data taken at different viewpoint positions, one image data is a reference image, and the other image data is a search image.
An area dividing step of dividing the reference image into small areas according to image characteristics;
A region boundary detection step for detecting a boundary line of each small region divided by the region division step;
A correspondence between a plurality of points on the boundary line of each small region detected by at least the region boundary line detection step between the reference image and the search image is obtained, and a parallax amount of the plurality of points on the boundary line of each small region is obtained. A parallax amount estimation step to be estimated;
A parallax amount frequency distribution creating step for creating a frequency distribution of parallax amounts at a plurality of points on the boundary estimated by the parallax amount estimating step for each small region;
A parallax amount frequency distribution analyzing step of analyzing the frequency distribution of the parallax amount of each small region created by the parallax amount frequency distribution creating step;
A region parallax amount estimation step for estimating a parallax amount of each small region based on the result analyzed by the parallax amount frequency distribution analysis step;
A parallax amount frequency distribution correction step for correcting the frequency distribution of the parallax amount of a small region where the boundary line overlaps with the small region whose parallax amount is estimated by the region parallax amount estimation step;
A region integration step of integrating the small regions according to the degree of similarity based on the parallax amount of each small region estimated by the region parallax amount estimation step and the color information of each small region;
A storage medium for storing a computer program for executing an area division integration method characterized by comprising: The parallax amount frequency distribution analysis step calculates a parallax amount average value, variance, and maximum frequency parallax amount for each small region from the frequency distribution of the parallax amount of each small region;
The region parallax amount estimation step is performed in order from a small region in which the average value of the parallax amount is relatively high in the small region in the image and the variance of the parallax amount is relatively small in the small region in the image. , Estimating the amount of parallax of each small region by using the parallax amount of the maximum frequency of each small region as the amount of parallax of the small region,
The parallax amount frequency distribution correcting step removes the parallax amount of the portion where the boundary line overlaps from the parallax amount frequency distribution of the small region where the boundary line overlaps with the small region where the parallax amount is estimated by the region parallax amount estimating step. The storage medium according to claim 28. The parallax amount frequency distribution creating step creates the frequency distribution by changing the contribution of the parallax amount of each point to the frequency distribution according to the image feature amount of each point on the boundary line of each small region. The storage medium according to any one of claims 26 to 29.

Images