JP2001243451A - Inter-image correspondence searching method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an inter-image correspondence searching method which can efficiently find close correspondence by a more accurate method with higher reliability. SOLUTION: This inter-image correspondence searching method which searches for correspondence between images having different viewpoints has an input step for inputting the images having the different viewpoints, a classification step for classifying the pixels or areas of the inputted images by properties, and a search step for finding the similarities by the pixels or areas between the inputted images and searching for corresponding pixel pair or area pair.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an inter-image correspondence search method for searching for a correspondence between a plurality of images having different viewpoints.

[0002]

2. Description of the Related Art There have been proposed many methods for finding a corresponding pixel or area between a plurality of images having different viewpoints. Among them, an area for which a correspondence is obtained by using local information near a pixel or an area is proposed. The base matching method is often applied.

Conventionally, the following two methods are known among the proposed methods. The first method (Qian Chen & Gera
rd Medioni, “A Volumetric Stereo Matching Method: A
pplication to Image-Based Modeling, ”Proceedings o
n Computer Vision and Pattern Recognition Conferen
ce '99, Vol. 1, pp. 29-34, June 1999), calculate the normalized correlation between the gray value near the target pixel and the gray value near the pixel of the other image to be searched for similarity. A degree map is created, and a correspondence with a particularly high degree of similarity among them is defined as a "seed correspondence". Next, the similarity is checked with respect to the neighborhood, and if the similarity is equal to or more than a certain value, the correspondence is set as a new seed correspondence, and the same processing is performed for the new seed correspondence, thereby generating a new seed correspondence. Then, a correspondence map is created.

In the second method (Graphics Communication Technology, Inc., "Distance Detection Method and Apparatus", Japanese Patent Laid-Open No. 2-268515), an image is divided into blocks and each block is dealt with. Regarding the method of obtaining, by calculating the amount of change in the gray value of the pixel in the block, comparing the amount of change with a predetermined threshold, and determining whether to search for correspondence based on the magnitude, This is a method of realizing high reliability and high speed of correspondence search.

[0005]

However, in the above-mentioned first method, there is a problem that a correspondence map is erroneous because a correspondence between pixels which cannot be dealt with is searched for.

In the above-mentioned second method, 1) since a correspondence is obtained for each block, it cannot be used when a close correspondence is desired.

[0007] 2) Since the amount of change in the gray value is treated only as an index for determining whether or not to perform a correspondence search, even if the correspondence is unreliable, it is treated in the same way as a highly reliable correspondence. There was a problem that reliability was lowered as a whole.

The present invention has been made in view of such a problem, and an object of the present invention is to provide an image capable of efficiently obtaining a close response by a more accurate and more reliable method. An object of the present invention is to provide an inter-correspondence search method.

[0009]

According to a first aspect of the present invention, there is provided an image correspondence search method for searching for a correspondence between a plurality of images having different viewpoints. An input step of inputting, for each of the plurality of input images, a classification step of classifying pixels or regions in the image according to a predetermined property, and using the result of the classification, each pixel or each pixel between the input images. And searching for a corresponding pixel pair or region pair by obtaining the similarity for each region.

According to a second aspect of the present invention, in the method for searching for correspondence between images according to the first aspect, the search step comprises the steps of:
A step of obtaining a pixel pair or a region pair having high correspondence reliability as an initial type correspondence is used, and a corresponding pixel pair or area pair is searched using the initial type correspondence.

According to a third invention, in the method for searching for correspondence between images in the first invention, the step of using the result of the classification in the search step includes processing a pixel or an area selected based on the result of the classification. It is to be excluded from the target.

[0012]

Embodiments of the present invention will be described below in detail with reference to the drawings.

(First Embodiment) First, a first embodiment of an inter-image correspondence search method according to the present invention will be described. FIG. 1 is a configuration diagram for explaining a first embodiment of the present invention. The big flow of the algorithm of this embodiment is as follows.

(1) First, a plurality of images D1 having different viewpoints are generated by the image input means 1.

(2) Next, in the image area classification means 2,
For each of the plurality of images D1, pixels or regions in the images are classified according to their properties to generate classification information D2.

(3) At the same time, in the similarity calculating means 3,
The similarity D3 for each pixel or region between input images is calculated in consideration of the classification information D2.

(4) Next, the correspondence searching means 4 searches for a corresponding pixel pair or area pair based on the similarity D3 obtained by the similarity calculating means 3 while considering the classification information D2. Find D4.

The overall device configuration includes, for example, a plurality or one camera and an image processing apparatus for processing a plurality of images D1 captured by the camera. Here, as a method of using the classification information D2, for example, the classification information D2 is used to determine whether a correspondence can be accurately obtained in principle in a correspondence search. As an example, if the vicinity of the target pixel is completely uniform, it is impossible to search for correspondence by area matching in principle. Therefore, such a region should be excluded in advance based on the classification information D2. As a result, erroneous responses can be prevented, and the search time can be reduced. As such classification information D2, variance of density values near the target pixel can be used. Details will be described later.

Hereinafter, each component shown in FIG. 1 will be described in detail. In the following, two images will be described from two different viewpoints for simplicity, but it is clear that the present invention is not an invention subject to such limitations.

First, the image input means 1 will be described. The image input unit 1 stores and outputs, in a two-dimensional array, a group of images captured by a digital camera or the like from two different viewpoints or a group of images subjected to appropriate preprocessing for searching for correspondence, or If the image is a color, the image is stored and output in a two-dimensional array for each color. Here, shooting from different viewpoints may be shooting by a plurality of cameras arranged at different positions, or may be shooting by moving to a different position with one camera. The appropriate processing here includes, for example, rectification and processing such as a smoothing filter and a median filter for removing noise added during image capturing.

FIG. 2 shows two images before rectification (FIGS. 2A and 2B) and two images (FIGS. 2C and 2D) obtained in the processing. This is a schematic representation. Here, rectification means an operation of making images taken from different viewpoints horizontal based on the camera calibration parameters, making epipolar lines of the left and right pixels horizontal and matching each other, For example, see the document Emanuele Trucco, Alessandr
o Verri, “Introductory Techniques for 3-D Computer
The details are described in Vision, "Prentice-Hall, New Jersey, USA, 1998", and the description thereof is omitted here.

Next, the image area classifying means 2 will be described. The image area classifying means 2 receives the two-dimensional array of the image group from the image input means 1 as input, and scans the two-dimensional array to calculate an appropriate property for each pixel by using information on the vicinity of the target pixel. Then, the properties are stored in a two-dimensional array and output as classification information D2.

For example, consider a pixel in a square of ((2M + 1) × (2M + 1)) (here, M is a positive integer) centered on the pixel of interest as a pixel neighborhood. The variance of the gray value is calculated for the vicinity of the pixel, and if the variance is smaller than a certain threshold value, 0 is stored, and if it is larger, 1 is stored in a two-dimensional array and output as classification information D2. As the pixel neighborhood here, it is conceivable to use a square, a rectangle, a circle centered on the pixel of interest, or a different area for each pixel, but it is clear that how to determine the pixel neighborhood does not limit the present invention. It is. As the classification information D2 and the method of using it, for example, the following can be considered. However, it is clear that the classification information D2 and the kind of the method of use thereof do not limit the present invention.

(1) As classification information D2 for measuring the uniformity of an image, for example, the variance of the density value near the target pixel, the maximum change width, the number of pixels whose deviation from the average is larger than a threshold value, The variance of the density derivative or the second derivative,
The maximum change width and the number of pixels whose deviation from the average is larger than a certain threshold are stored in a two-dimensional array. Then, for example, 1) if the uniformity is higher than a certain threshold, it is determined that a reliable similarity D3 cannot be obtained, and a similarity calculation or a correspondence search is not performed from the beginning. . As a result, erroneous correspondence can be prevented, and similarity calculation or correspondence search can be speeded up.

2) If the uniformity is low, that is, if the neighboring pixels are complicated, if a correspondence is found, it can be said that the reliability of the correspondence is high. Can also be used.

3) The degree of change in parallax can be measured using the variance of the derivative of the density value, the maximum change width, the number of pixels whose deviation from the average is larger than a threshold value, and the like.
For example, if these values are small to some extent, the density value changes smoothly within that region, and there is a high possibility that the change in parallax is small. It is thought that there is.

4) In the method of calculating the similarity by ignoring the average in the vicinity of the target pixel as in the similarity calculation method using the normalized correlation, an image region in which the pixel value changes linearly ( It is very likely that an erroneous response will occur in the gradation area). Therefore, the reliability of the similarity can be increased by checking the uniformity, for example, the variance of the differential value of the density value, detecting the gradation area, and not calculating the similarity in that area.

Such a method of utilization can be considered. However, the type of the classification information D2 for measuring the uniformity of the image,
Clearly, its use does not limit the invention. The derivative of the density value and the second derivative here are
The following can be considered. The coordinates of the pixel of interest are (i, j),
If the density value is P (i, j) and the differential value is D (i, j),

[0029]

(Equation 1)

## EQU1 ## _DV and _DH are in the i direction and j, respectively.
This is the result of the Sobel filter in the direction. Or
A method using D _V and D _H is also conceivable. As for the secondary differentiation, the primary differentiation is performed once again on the image obtained by performing the primary differentiation. However, it is clear that the method of differentiation does not limit the present invention. The maximum change width here is the difference between the minimum value and the maximum value of the density value or its differential value in the vicinity of the target pixel.

(2) Instead of using the density value of a pixel,
For example, for a pixel of a color represented by RGB, each R
The variance is calculated for each of the GBs, and the color having the highest variance is stored as a two-dimensional array. For example, in the similarity calculation, a similarity calculation may be performed for each color, and a value obtained by weighting the color with the highest variance and performing weighted averaging as the similarity D3 may be used. As a result, a more reliable response can be obtained. However, it is clear that the method of expressing the color, the type of the classification information D2 obtained therefrom, and the method of using the same do not limit the present invention.

(3) The average value in the vicinity of the target pixel is stored as a two-dimensional array. Then, for example, in accordance with the average size, it is determined whether to calculate the similarity D3 for the target pixel, to execute the correspondence search, or to determine its reliability. As a result, for example, in a black region, the dispersion is not small due to fluctuation due to the CCD or the like, but the region in which the dispersion itself is not reliable can be eliminated in advance, thereby improving the reliability of the response.

Next, the similarity calculating means 3 will be described.
The similarity calculating unit 3 receives an image group (a plurality of images D1) from the image input unit 1 and classification information D2 from the image region classification unit 2, and pays attention to one of the images while considering the classification information D2. The similarity D3 is calculated using the gray value of the neighborhood of the pixel and the neighborhood of the pixel for which it is desired to calculate the similarity D3 of the other image, and the similarity D3 is calculated in the vertical and horizontal directions (u, v) and the parallax direction (d) of the image. It is output as a three-dimensional array.

FIG. 3 illustrates the three-dimensional array. For example, on the basis of the left image, a pixel R (i, j) whose coordinates of the right image is (i + k, j) for a pixel L (i, j) whose coordinates of the left image is (i, j). -K, j)
A similarity D3 with (0 <k <D) is calculated and stored. The pixel neighborhood here is not limited to a square, rectangle, or circle centered on the pixel of interest, and it is conceivable to use a different area for each pixel. However, how to determine the pixel neighborhood does not limit the present invention. Is obvious.

[0035]

(Equation 2)

It is conceivable to use any one of the numbers of differences equal to or larger than the threshold value or to adaptively switch and use the plurality of numbers for each pixel. However, the similarity D3 is not limited to the number described above.

As a method of calculating the similarity D3, for example, instead of directly calculating the two images for which the similarity D3 is to be calculated, the similarity D3 is roughly calculated using a reduced or blurred image. A method using a so-called multi-resolution analysis in which a region in which the similarity D3 is considered to be good is calculated, and then the similarity D3 is calculated for details (see "Sharp Corporation, Japanese Patent Laid-Open No. “Stereo correspondence search device” ”).

Next, the correspondence search means 4 will be described. Correspondence search means 4 receives classification information D from image area classification means 2
2 and a similarity D3 from the similarity calculating means 3 to search for a correspondence and output a correspondence map. For example, the pixel of interest of the reference image is associated with the pixel most likely to correspond in the other image in consideration of the similarity and the reliability of the similarity in the other image. The reliability of the similarity here is described in the description of the type of the classification information in the image area classification means 2 and the method of using the classification information, and thus the description is omitted here.

The correspondence map referred to here is, for example, a disparity map (plotted on the left image, indicating how much the pixel on the right image corresponding to the target pixel of the left image is shifted with respect to the left image. Two-dimensional array). However, it is clear that the method of outputting the correspondence map does not limit the present invention.

(Second Embodiment) Hereinafter, a second embodiment of the method for searching for correspondence between images according to the present invention will be described. FIG. 4 is a configuration diagram for explaining the second embodiment of the present invention. The configuration of the second embodiment is obtained by adding the initial type correspondence generation unit 5 to the configuration of the first embodiment. Hereinafter, a description will be given of the newly provided initial type correspondence generation means 5 and a detailed description of a specific method of using the initial type correspondence D5 generated by the initial type correspondence generation means 5.

The initial type correspondence generating means 5 receives the similarity D3 from the similarity calculating means 3 and the classification information D2 from the image area classifying means 2 as input and performs correspondence based on the classification information D2 and the similarity D3 map. Is stored as an initial type correspondence D5 and its coordinates and disparity are stored and output in the form of a three-dimensional vector. Hereinafter, a specific generation method for the initial type will be described in detail.

(1) The similarity D3 is scanned in the parallax direction in the three-dimensional array of the similarity D3, the number of the local maximums is counted, and a pixel having only one local maximum is obtained.

(2) A pixel having a maximum similarity D3 greater than a certain threshold value among the pixels obtained in (1) is obtained.

(3) The reliability corresponding to the pixel obtained in (2) is calculated.

(4) The pixels obtained in (2) are sorted according to the corresponding reliability.
The coordinates and the parallax are stored in the form of a three-dimensional vector with 0 as the initial type correspondence D5.

Here, the correspondence reliability means, for example, when a pixel corresponding to a target pixel of one image is searched for in a certain region of another image, the similarity D3 and the similarity D3 are present in that region. This is a value calculated from the uniqueness and width of the peak of the similarity D3 for the pixel and the classification information D2 of the pixel of interest. The method using the classification information D2 has been described in detail in the description of the image area classification means 2 of the first embodiment, and will not be described here. As a method of use, for example, if the width of the peak of the similarity D3 is described, the wider the width is, the less reliable the peak is. There is one that is not adopted as the initial type correspondence D5. However, it is clear that the method of calculating the corresponding reliability does not limit the present invention.

FIG. 5 schematically shows a search for a correspondence using the initial kind correspondence. The search for the correspondence will be described in detail with reference to FIG. Here, for simplicity, a case will be described in which only the similarity D3 is used to search for a correspondence, but this search method does not limit the present study.

(1) First, in the three-dimensional array of the similarity D3, attention is paid to a pixel in the positive direction of u from the kind-correspondence already obtained by the initial kind correspondence generating means 5.

(2) Check the parallax of the pixel of interest and compare several parallaxes close to the parallax of the current coordinates to determine the similarity D3
Is the maximum and the similarity D3 is larger than a certain threshold, the disparity is determined to be the disparity of the coordinates, and the coordinates and the disparity are stored as a three-dimensional vector corresponding to the seed.

(3) This is performed in the remaining three directions (negative direction of u,
(positive direction of v, negative direction of v).

(4) The current species correspondence is stored in the correspondence map as the correspondence, and the process proceeds to the next species correspondence and returns to (1). This step is repeated until there is no seed correspondence. Here, in step (2), the disparity having the maximum similarity D3 is determined as the disparity of the pixel of interest. However, the reliability of the similarity D3 using the uniformity in the description regarding the image area classification unit 2 of the first embodiment is determined. For example, it is conceivable that the disparity that maximizes (similarity D3 + reliability) is set as the disparity of the pixel of interest.
The method of using the classification information D2 is described in detail in the image area classification means 2.

In the second embodiment, a special use of the classification information D2 can be considered, which will be described. First, the gradient of the pixel of interest is stored as a vector in a two-dimensional array. And according to the direction and strength of the gradient,
For example, it is conceivable to change a threshold value used for a correspondence search of a pixel adjacent to the target pixel. For example, when there is a strong gradient in the adjacent direction, by changing the threshold value so as to suppress the correspondence of the pixel from being a seed correspondence, an area in which the parallax may have changed may be used. False responses can be reduced. Here, the gradient of the pixel of interest may be as follows. The coordinates of the pixel of interest are (i, j), the density value is P (i, j), and the gradient is G (i, j).
j)

[0053]

(Equation 3)

Is as follows. However, it is clear that the method of calculating the gradient does not limit the present invention.

(Third Embodiment) Hereinafter, a third embodiment of the method for searching for correspondence between images according to the present invention will be described. FIG. 6 is a configuration diagram for explaining the third embodiment of the present invention. In the third embodiment, the similarity calculation execution determining means 301 and the similarity calculation execution means 30 are included in the similarity calculation means 3 of the first embodiment.
2 is provided. Similarity calculation execution determination means 301
Then, it is determined whether or not the similarity calculation should be performed on the target pixel based on the classification information D2 by inputting a plurality of images D1. The similarity calculation execution means 302 executes the similarity calculation according to the determination result at this time. The type and use method of the classification information D2 used for the determination have been described in detail in the image area classification unit 2 of the first embodiment, and thus will not be described here.

(Fourth Embodiment) Hereinafter, a fourth embodiment of the inter-image correspondence search method according to the present invention will be described. FIG. 7 is a configuration diagram for explaining the fourth embodiment of the present invention. In the fourth embodiment, an adjacent correspondence judging means 401 and a kind correspondence storing means 402 are provided in the correspondence searching means 4 of the second embodiment. In the adjacent correspondence determination unit 401, the classification information D2
And the similarity D3, the reliability of the initial type correspondence D5 and the correspondence adjacent to the type correspondence D402 from the type correspondence storage unit 402 is determined.

The type correspondence storage means 402 stores the type of correspondence determined as reliable by the adjacent type determination means 401 as a new type correspondence D401. The type, use method, or use method of the similarity of the classification information D2 used for the determination are described in detail in the second embodiment, and will not be described here.

(Fifth Embodiment) Hereinafter, a fifth embodiment of the method for searching for correspondence between images according to the present invention will be described. FIG. 8 is a configuration diagram for explaining the fifth embodiment of the present invention. In the fifth embodiment, the correspondence search means 4 of the second embodiment is provided with an adjacent correspondence determination means 401, a kind correspondence storage means 402, a correspondence reliability calculation means 403, and a correspondence re-search means 404. The details of the adjacent correspondence determination means 401 and the kind correspondence storage means 402 have been described in the fourth embodiment, and will not be described here.

The correspondence reliability calculating means 403 calculates the reliability of the obtained correspondence. Specifically, for example, occlusion (occlusion) always occurs in a region where the parallax is large. If there is a correspondence in an area where occlusion should have occurred, it can be determined that the correspondence is not reliable. The occlusion here is an area where only one image is captured between two images. The occlusion is mainly the area where the object in front is moved due to parallax and is in the shadow of the object. Alternatively, in consideration of the classification information D2, the reliability of the correspondence may be calculated using the reliability of the similarity described in the first embodiment. However, it is clear that the method of calculating the corresponding reliability does not limit the present invention.

(Sixth Embodiment) Hereinafter, a sixth embodiment of the method for searching for correspondence between images according to the present invention will be described. FIG. 9 is a configuration diagram for explaining the sixth embodiment of the present invention. In the sixth embodiment, a similarity calculation reliability determination unit 4011 and an adjacent correspondence reliability calculation unit 4012 are provided in the adjacent correspondence determination unit 401 of the fourth or fifth embodiment.

In the similarity calculation reliability determination means 4011,
The reliability of the calculation of the similarity calculated based on the classification information D2 and the similarity D3 is calculated. The method of calculating the reliability of the similarity calculation using the classification information D2 has been described in detail in the description of the classification information D2 in the first embodiment, and thus will not be repeated here. As a method of calculating the reliability of the similarity calculation using the similarity D3, for example, the similarity D3 is scanned in the parallax direction in the above-described three-dimensional array of the similarities D3, and the sum of the similarities is obtained. If the sum of the similarities is large, it is highly likely that there are many pixels similar to the pixel of interest, and thus it is determined that the reliability is low. However, it is apparent that the type of the method of calculating the reliability of the similarity calculation does not limit the present invention.

The adjacent correspondence reliability calculating means 4012 calculates the initial kind correspondence D5, the similarity D3, and the reliability D4 of the similarity calculation.
011 Based on the classification information D2, the reliability of the correspondence adjacent to the correspondence of interest is determined. The method of determining the reliability of the correspondence has been described in detail in the second embodiment, and will not be described here.

A. An invention having the following procedure is extracted from the specific embodiment described above.

1. An image correspondence search method for searching for a correspondence between a plurality of images having different viewpoints, comprising: an input step of inputting a plurality of images having different viewpoints; and for each of the plurality of input images, a pixel or a region in the image. And a search step of searching for a corresponding pixel pair or region pair by obtaining a similarity between each pixel or each region between the input images by using the result of the classification. A method for searching for correspondence between images, characterized by comprising:

2. The search step includes a step of obtaining a pixel pair or a region pair having a high correspondence reliability as an initial type correspondence from the result of the classification and the similarity obtained in the search step, and utilizing this initial type correspondence. , Searching for a corresponding pixel pair or region pair. 2. The method for searching for correspondence between images according to item 1.

3. The use of the result of the classification in the search step is to exclude a pixel or a region selected based on the result of the classification from processing targets. 2. The method for searching for correspondence between images according to item 1.

4. 2. The predetermined property in the classification step is a property calculated based on a physical quantity of each pixel or each area and its vicinity. 2. The method for searching for correspondence between images according to item 1.

5. 2. The predetermined property in the classification step is a property calculated based on a texture parameter of each pixel or each area and its vicinity. 2. The method for searching for correspondence between images according to item 1.

6. The predetermined property in the classification step is a property that can be used when searching for a corresponding pixel pair or area pair. 2. The method for searching for correspondence between images according to item 1.

7. 5. The predetermined property in the classification step is a property calculated based on the physical quantity of each pixel or each area and its vicinity. 2. The method for searching for correspondence between images according to item 1.

8. 5. The predetermined property in the classification step is a property calculated based on a texture parameter of each pixel or each area and its vicinity. 2. The method for searching for correspondence between images according to item 1.

9. 2. The predetermined property in the classification step is a property that can be used when searching for a corresponding pixel pair or area pair. 2. The method for searching for correspondence between images according to item 1.

10. The process of calculating the similarity in the search step includes a step of determining whether or not to calculate the similarity for each pixel or each region, and a step of calculating the similarity based on the determination result. 2. The method for searching for correspondence between images according to item 1.

11. In seeking the above initial species support,
It is characterized in that at least one of uniqueness of a peak of similarity, peak width of similarity, and a texture parameter near a pixel or an area is used. 2. The method for searching for correspondence between images according to item 1.

12. In the obtained initial type correspondence, a step of obtaining the reliability of the correspondence between the adjacent pixels or regions, a step of newly storing the correspondence recognized based on the reliability as a type correspondence, Determining the reliability of the correspondence between pixels or regions that are further adjacent to the type correspondence. 2. The method for searching for correspondence between images according to item 1.

13. The step of obtaining the reliability of the adjacent image or region is a step of obtaining the reliability of the obtained similarity using the classification result of the classification step, the similarity, and the reliability of the similarity, 11. using the classification result of the classification step to determine the reliability of an adjacent pixel or region. 2. The method for searching for correspondence between images according to item 1.

14. The reliability of the similarity is at least one of the uniqueness of the similarity peak, the peak width of the similarity, the second derivative of the similarity peak, the sum of the similarities, the texture parameter near the pixel of interest, and the noise variance. 13. It is characterized in that 2. The method for searching for correspondence between images according to item 1.

15. In the obtained initial type correspondence, a step of obtaining the reliability of the correspondence between the adjacent pixels or regions, a step of storing the correspondence recognized based on the reliability as a new type correspondence, and a step of the classification step A step of obtaining the reliability of the stored correspondence based on the classification result of the above and the similarity obtained in the search step; and a step of reexamining the correspondence based on the reliability of the obtained correspondence. Determining the reliability of the correspondence of the adjacent pixel or region with respect to the reconsidered correspondence. 2. The method for searching for correspondence between images according to item 1.

16. The step of obtaining the reliability of the adjacent image or region is a step of obtaining the reliability of the obtained similarity using the classification result of the classification step, the similarity, and the reliability of the similarity, A step of obtaining the reliability of the correspondence between adjacent pixels or areas using the classification result of the classification step.
5. 2. The method for searching for correspondence between images according to item 1.

17. The reliability of the similarity is at least one of the uniqueness of the similarity peak, the peak width of the similarity, the second derivative of the similarity peak, the sum of the similarities, the texture parameter near the pixel of interest, and the noise variance. 16. It is based on 2. The method for searching for correspondence between images according to item 1.

18. The reliability of the correspondence is the smoothness of the stored correspondence disparity, the second derivative of the stored correspondence disparity, the uniqueness of the similarity peak, the peak width of the similarity, and the vicinity of the pixel of interest. 1 based on at least one of the texture parameters
6. 2. The method for searching for correspondence between images according to item 1.

19. In an inter-image correspondence search method for searching for a correspondence between a plurality of images having different viewpoints, an image input step of inputting a plurality of images having different viewpoints, and an intra-image processing for each of the plurality of images obtained in the image input step. An image region classification step of classifying pixels or regions of each of them according to their properties to generate classification information, and calculating the similarity of each pixel or region between input images based on the classification information obtained in the image region classification step. A similarity calculation step for calculating, a classification search obtained in the image area classification step, and a corresponding search for searching for a corresponding pixel pair or area pair based on the similarity obtained in the similarity calculation step. And a step of searching for correspondence between images.

20. A classification information obtained in the image region classification step, and an initial type correspondence generation step of obtaining a pixel pair or a region pair having high correspondence reliability as an initial type correspondence based on the similarity obtained in the similarity calculation step. 18. The method according to claim 19, wherein a corresponding pixel pair or region pair is searched in the correspondence search step using the initial kind correspondence. 2. The method for searching for correspondence between images according to item 1.

21. 18. The image area classification step is characterized in that the image area is classified based on a property serving as a clue for determining whether or not to calculate the similarity in the similarity calculation step. 2. The method for searching for correspondence between images according to item 1.

22. 20. A property that is a clue for determining whether or not to calculate the similarity in the image area classification step is calculated based on the vicinity of the target pixel. 2. The method for searching for correspondence between images according to item 1.

23. 20. The property which is a clue for determining whether or not to calculate the similarity in the image area classification step is a texture parameter near the target pixel. 2. The method for searching for correspondence between images according to item 1.

24. 18. The image area classification step is characterized in that classification is performed in the correspondence search step based on a property serving as a clue for search. 2. The method for searching for correspondence between images according to item 1.

25. 23. A property serving as a clue for search in the image area classification step is calculated based on the vicinity of a target pixel. 2. The method for searching for correspondence between images according to item 1.

26. 23. The characteristic that is a clue for the search in the image area classification step is a texture parameter near the pixel of interest. 2. The method for searching for correspondence between images according to item 1.

27. The image area classification step classifies based on a property that is a clue for determining whether or not to calculate the similarity in the similarity calculation step, and based on a property that is a clue for the search in the correspondence search step. 19. characterized in that: 2. The method for searching for correspondence between images according to item 1.

28. The similarity calculation step is a similarity calculation execution determination step of determining whether to execute similarity calculation based on the classification information obtained in the image area classification step, and the similarity calculation execution determination step 18. a similarity calculation execution step of calculating similarity based on the determination result in 2. The method for searching for correspondence between images according to item 1.

29. The initial type correspondence generation step includes
1) uniqueness of the similarity peak obtained in the similarity calculation step, or 2) width of the similarity peak obtained in the similarity calculation step, or 3) texture parameter near the pixel of interest. 19. A method for generating a highly reliable correspondence by using one of them. 2. The method for searching for correspondence between images according to item 1.

30. The correspondence search step is based on the classification information obtained in the image area classification step and the similarity obtained in the similarity calculation step, and is adjacent to the initial type correspondence obtained in the initial type correspondence generation step. An adjacent correspondence determining step of calculating the correspondence reliability of the pixel or region pair to be executed, and a seed correspondence storing step of storing the correspondence determined to correspond by the adjacent correspondence determining step as a new seed correspondence. 20. The kind correspondence stored in the kind correspondence storing step is further similarly determined in the adjacent correspondence determining step. 2. The method for searching for correspondence between images according to item 1.

31. The adjacent correspondence determination step is a similarity calculation reliability calculation step of calculating the reliability of the calculation of the similarity obtained in the similarity calculation step while considering the classification information obtained in the image area classification step, Based on the reliability of the calculation of the similarity obtained in the similarity calculation reliability calculation step, the classification information obtained in the image area classification step, and the similarity obtained in the similarity calculation step 30. An adjacent correspondence reliability calculating step of calculating the correspondence reliability of a pixel or an area pair adjacent to the kind correspondence. 2. The method for searching for correspondence between images according to item 1.

32. The reliability of the similarity calculation obtained in the similarity calculation reliability calculation step is 1) the uniqueness of the peak of the similarity obtained in the similarity calculation step, or 2) the uniqueness of the similarity calculation step. The width of the similarity peak obtained, or 3) the second derivative of the similarity peak obtained in the similarity calculation step, or 4) the sum of the similarity obtained in the similarity calculation step, or 5) the notice. 31. It is at least one of a texture parameter near a pixel and 6) a variance of noise. 2. The method for searching for correspondence between images according to item 1.

33. The correspondence search step is adjacent to the initial type correspondence obtained in the initial type correspondence generation step based on the classification information obtained in the image area classification step and the similarity obtained in the similarity calculation step. An adjacent correspondence determining step of calculating a correspondence reliability of a pixel or an area pair; a seed correspondence storing step of newly storing a correspondence determined to correspond by the adjacent correspondence determining step as a seed correspondence; and the image area step. A correspondence reliability calculation step of calculating the reliability of the correspondence stored in the type correspondence storage step based on the classified information obtained and the similarity obtained in the similarity calculation step; and a correspondence reliability calculation step. Re-searching step for searching again for correspondence based on the reliability of the correspondence. 19, characterized in that to continue to determine at further Similarly the adjacent corresponding determining step with respect to the species corresponding stored at-flop. 2. The method for searching for correspondence between images according to item 1.

34. The adjacent correspondence determination step is a similarity calculation reliability calculation step of calculating the reliability of the calculation of the similarity obtained in the similarity calculation step while considering the classification information obtained in the image area classification step, Based on the reliability of the calculation of the similarity obtained in the similarity calculation reliability calculation step, the classification information obtained in the image area classification step, and the similarity obtained in the similarity calculation step 33. An adjacent correspondence reliability calculating step of calculating the correspondence reliability of a pixel or an area pair adjacent to the kind correspondence. 2. The method for searching for correspondence between images according to item 1.

35. The reliability of the similarity calculation obtained in the similarity calculation reliability calculation step is 1) the uniqueness of the peak of the similarity obtained in the similarity calculation step, or 2) the uniqueness of the similarity calculation step. The width of the similarity peak obtained, or 3) the second derivative of the similarity peak obtained in the similarity calculation step, or 4) the sum of the similarity obtained in the similarity calculation step, or 5) the notice. 34. It is at least one of a texture parameter near a pixel and 6) a variance of noise. 2. The method for searching for correspondence between images according to item 1.

36. The reliability of the correspondence obtained by the correspondence reliability calculation step is 2) the smoothness of the corresponding parallax stored in the species correspondence storage step, or 2) the corresponding parallax stored in the species correspondence storage step. Second derivative,
Or 3) the uniqueness of the similarity peak obtained in the similarity calculation step, or 4) the width of the similarity peak obtained in the similarity calculation step, or 5) the texture parameter in the vicinity of the target pixel. 34. characterized by at least one 2. The method for searching for correspondence between images according to item 1.

B. The invention described below 19. ~ 36.
Effects and corresponding embodiments will be described.

19. According to the similarity calculation in the similarity calculation step and the correspondence search in the correspondence search step, by using not only the vicinity of the target pixel but also the classification information obtained in the image area classification step,
High efficiency of the similarity calculation and high reliability of the correspondence can be realized.
Also, 19. Corresponds to the first embodiment described above.

20. According to the above, when performing a correspondence search, a correspondence with high correspondence reliability is obtained as a seed correspondence by a configuration initial kind correspondence generation step based on the classification information obtained in the image area classification step, and based on the seed correspondence. By performing the correspondence search, high reliability of the correspondence can be realized. Also, 2
0. Corresponds to the above-described second embodiment.

21. Corresponds to the first embodiment described above.

22. Corresponds to the first embodiment described above.

23. Corresponds to the first embodiment described above.

24. Corresponds to the first embodiment described above.

25. Corresponds to the first embodiment described above.

26. Corresponds to the first embodiment described above.

27. Corresponds to the first embodiment described above.

28. According to the method, a similarity calculation execution determination step determines whether or not a reliable similarity can be obtained based on the classification information obtained in the image region classification step, and as a result, a reliable similarity By calculating the similarity in the similarity calculation execution step only for the pixels or regions for which it has been determined that can be calculated, the calculation of the similarity is speeded up. Further, since the correspondence search is not executed for the unreliable similarity, erroneous correspondence is reduced in the initial type correspondence generation step and the correspondence search step. 28. Corresponds to the third embodiment described above.

29. Corresponds to the third embodiment described above.

30. According to the above, the correspondence between the pixel or region pair adjacent to the initial type correspondence obtained in the initial type correspondence generation step based on the classification information obtained in the image area classification step and the similarity obtained in the similarity calculation step The reliability is calculated, the correspondence is determined based on the reliability, stored as a new species correspondence, the same processing is performed for the new species correspondence, and the correspondence is searched for in a propagation manner. Therefore, the reliability of the response in the vicinity of the initial type response is guaranteed, and the reliability of the response as a whole can be improved. Also, 3
0. Corresponds to the fourth embodiment described above.

31. According to the method, the reliability of the similarity is calculated in the similarity calculation reliability calculation step based on the classification information obtained in the image region classification step and the similarity obtained in the similarity calculation step, and the adjacent correspondence reliability is calculated. By searching for a correspondence in consideration of the reliability of the similarity in the calculation step, the reliability of the correspondence can be improved.
32. Corresponds to the sixth embodiment described above.

32. Corresponds to the sixth embodiment described above.

33. According to the classification information obtained in the image area classification step and the correspondence stored in the species correspondence storage step and the similarity obtained in the similarity calculation step, the correspondence reliability calculation step in the species correspondence storage step The reliability of the stored correspondence is calculated, and the correspondence is searched again in the correspondence re-searching step based on the reliability of the correspondence. Higher accuracy can be realized. 33. Corresponds to the fifth embodiment described above.

34. According to the method, the reliability of the similarity is calculated in the similarity calculation reliability calculation step based on the classification information obtained in the image region classification step and the similarity obtained in the similarity calculation step, and the adjacent correspondence reliability is calculated. Since the search is performed in consideration of the reliability of the similarity calculated in the calculation step, the corresponding high reliability can be realized. 34. Corresponds to the sixth embodiment described above.

35. Corresponds to the sixth embodiment described above.

36. Corresponds to the fifth embodiment described above.

[0119]

According to the present invention, it is possible to provide a method for searching for a correspondence between images, which can efficiently obtain a dense correspondence by a more accurate and more reliable method.

[Brief description of the drawings]

FIG. 1 is a configuration diagram for explaining a first embodiment of the present invention.

FIG. 2 is a diagram schematically showing two images before rectification and two images obtained in the processing.

FIG. 3 is a diagram showing how similarities D3 are arranged in three dimensions in the vertical and horizontal directions (u, v) and the parallax direction (d) of an image.

FIG. 4 is a configuration diagram for explaining a second embodiment of the present invention.

FIG. 5 is a diagram schematically illustrating a search for a correspondence using an initial kind correspondence.

FIG. 6 is a configuration diagram for explaining a third embodiment of the present invention.

FIG. 7 is a configuration diagram for explaining a fourth embodiment of the present invention.

FIG. 8 is a configuration diagram for explaining a fifth embodiment of the present invention.

FIG. 9 is a configuration diagram for explaining a sixth embodiment of the present invention.

[Description of Signs] 1 Image input means 2 Image area classification means 3 Similarity calculation means 4 Correspondence search means 5 Initial type correspondence generation means 301 Similarity calculation execution determination means 302 Similarity calculation execution means 401 Adjacent correspondence determination means 402 types correspondence Storage means 403 Correspondence reliability calculation means 404 Correspondence re-search means 4011 Similarity calculation reliability determination means 4012 Adjacent correspondence reliability calculation means

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 2F065 AA51 DD06 FF05 JJ03 JJ05 JJ26 QQ13 QQ24 QQ26 QQ27 QQ28 QQ33 QQ34 QQ38 QQ41 QQ42 5B057 BA11 CE05 DA20 DB06 DB09 DC02 DC32 5L096 CA05 EA06 FA32 FA33 FA03 FA03 FA03 FA03

Claims (3)

[Claims] 1. An image correspondence search method for searching for a correspondence between a plurality of images having different viewpoints, comprising: an input step of inputting a plurality of images having different viewpoints; And a classification step of classifying the pixels or regions in a predetermined property according to a predetermined property.Using the result of the classification, the similarity of each pixel or each region between the input images is obtained, and a corresponding pixel pair or region pair A search step for searching. 2. The method according to claim 1, wherein the searching step includes a step of obtaining a pixel pair or a region pair having high correspondence reliability as an initial type correspondence from the result of the classification and the similarity obtained in the searching step. 2. The method according to claim 1, wherein the correspondence is used to search for a corresponding pixel pair or region pair. 3. The method according to claim 1, wherein using the result of the classification in the search step includes excluding a pixel or an area selected based on the result of the classification from processing targets. Inter-image correspondence search method.