JPH1021403A - Method and device for extracting correspondent points - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a device for extracting correspondent points by which high-accuracy correspondent point extracting processing can be performed over all the area without discrimination of object or background in the image to be used. SOLUTION: A template image preparing part 106 prepares a template image by receiving template size data preserved in a data preserving part 105, template size data from a template size change part 109 or a reference image from a reference image memory 102 and an evaluation function operating part 107 operates an evaluation value between the reference image and the image to be searched preserved in an image memory 103 to be searched and outputs correspondent point candidates and their evaluation values.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a corresponding point extraction method and apparatus used for obtaining distance information of a subject.

[0002]

2. Description of the Related Art Conventionally, as a method of using distance information of a subject using an image, as shown in FIG.
Object 24c using more than one camera 24a, 24b
And a method using the parallax between the cameras 24a and 24b, and one camera 24d as shown in FIG.
There is known a method of photographing a subject 24c while moving in the direction of an arrow, and using a motion parallax between obtained time-series images.

In both of the methods shown in FIGS. 24 (a) and 24 (b), corresponding points are extracted in order to obtain a parallax required for obtaining a distance. In the method shown in FIG. 24A, the corresponding point extraction means that the left camera 2
The purpose is to find corresponding points at all points on the image obtained by 4a and the image obtained by the right camera 24b. In the method shown in FIG. 24 (b), corresponding points are obtained at all points on the image obtained at a certain position and the image obtained at the moved position. Note that the reason why the correspondence is obtained at all points in the image is that it is not possible to know where in the image the subject appears.

[0004] A template matching method is known as one of the representative methods of the corresponding point extraction processing method.
This template matching method is used to find a corresponding point in a reference image 25a as shown in FIG.
In this method, the template 25b surrounding the point P is considered, the template 25b is moved to the entire area on the image 25c to be searched, and the corresponding point is determined by calculating the similarity at each point.

As an evaluation function for obtaining the similarity, there are a function using a difference between luminance values as in the following equation (1) and a function using a correlation value between luminance values as in the following equation (2).

[0006]

(Equation 1)

[0007]

(Equation 2) In the above equations (1) and (2), F (i, j) represents a template, and A (i, j) represents an image to be searched.
That is, the above equations (1) and (2) show the similarity when the position of the template is moved by (x, y) on the searched image. In addition, using equation (1), E (x, y)
Is the corresponding point, and theoretically E (x,
The minimum value of y) is 0. Also, using equation (2),
The point where σ (x, y) is the maximum is the corresponding point, and the maximum value of σ (x, y) is 1 in theory.

In addition, as another evaluation function, a threshold value ε having a difference between each pixel in the template and one pixel on the image to be searched in the entire template as shown in the following equation (3).
There is a function that obtains the following number of pixels and uses the point having the largest number of pixels as the corresponding point.

[0009]

(Equation 3) Using equation (3), the point where C (x, y) is the maximum is the corresponding point, and the maximum value is theoretically the total number of pixels in the template.

Conventionally, the amount of parallax at each point in an image is obtained by using the corresponding point extraction method as described above, and the triangulation rule is calculated using the amount of parallax, the focal length of the camera, and the position information of the camera. The distance information of the subject is obtained from the image by using this method.

[0011]

However, according to the template matching method shown in the above-mentioned conventional example, when a corresponding point extraction process is performed between images obtained by capturing an object at a finite distance from a camera, the object is not identified. There is a problem that the extraction accuracy is greatly reduced in a certain part.

FIG. 26 is a diagram showing this. FIG. 24A shows an image of a car taken as a subject 24c in an imaging system having two cameras 24a and 24b shown in FIG. 24A, and FIG. 26A is obtained by the left camera 24a. 26 (b) and 26 (c) show the right camera 24.
Each image obtained by b is shown. For convenience of explanation, it is assumed that a car is a subject, a house or a tree is a background, and a house or a tree is far from the car.

Here, based on the image shown in FIG.
Consider cutting out a template. At this time, the template needs to have a certain size.
The reason is that the brightness value of the image around the point A and the point B of the car in the image of FIG. I do. In addition, there is a similar luminance distribution area around point C where the leaves of the tree overlap. In order to accurately extract corresponding points even in such an area, the size of the template needs to be a certain size.

For example, if the size of the template is determined from the point B in the image of FIG. 26A to the size shown in the figure, tires and the like enter the template 261 in addition to the door. Can be determined.

Next, consider the point D at the end of the car, which is the subject in the image of FIG. When the template 262 is created at the point D at the same template size as the point B, the template 262 includes a house as a background in addition to a car as a subject. When the corresponding points are extracted using the template 262, an erroneous correspondence occurs in which an erroneous point F is a corresponding point instead of the correct point E as shown in FIG. 26C.

This is because when the evaluation function of the corresponding point at the point E is obtained, the part of the car which is the subject is good, but the background part is different, so that the value of the evaluation function in that area becomes bad, Is not a corresponding point. Also,
When the evaluation function of the corresponding point is obtained at the point F, the value of the evaluation function in the background is the same and the area is good, and the car part which is the subject is different, but happens to have the same luminance value. This is caused by the fact that the value of the evaluation function in the area is improved because the number of areas is large, and the point F is recognized as the corresponding point. Such an erroneous response is a phenomenon caused by a difference in the amount of parallax due to a car being a subject being in front of a house or a tree being a background, and therefore, a subject existing at a finite distance is imaged. In some cases, the corresponding point extraction accuracy is degraded in a neighboring region where the subject and the background are in contact, or in a region where the distance greatly changes even in the subject.

As described above, when the template matching method is used for the corresponding point extraction processing, the size of the template to be used must be increased to some extent, and the corresponding points between images obtained by capturing an object at a finite distance are taken. If the extraction process is performed using a template having a certain size, there is a problem that the accuracy of extracting corresponding points is deteriorated in a neighboring region where the subject and the background are in contact.

The evaluation function expressed by the above equation (3) is:
This is effective to some extent in preventing the corresponding point extraction accuracy from deteriorating in the vicinity area where the subject is in contact with the background. However, when most of the background is included in the template,
The number of pixels whose difference value from one pixel on the image searched for by the pixels in the template is smaller than a certain threshold ε is larger when the background is drawn on the template, and the corresponding point extraction accuracy is deteriorated. There was a problem of doing.

Also, an image obtained by the method shown in FIGS. 24A and 24B has an area called an occlusion area having no corresponding point, and it is difficult to detect the area. There was a problem.

For example, the point corresponding to point I in the image of FIG. 12A is hidden by the subject car in the image of FIG. 12B, and does not exist in this image. Such an area is called an occlusion area. Since there is no corresponding point at a point in this area, a result of a point in this area must not be used as a corresponding point extraction result. As a method for detecting this area, when a corresponding point candidate in the reference image is obtained in the image to be searched, a template is created in the image to be searched centering on that point, and the template is used as a reference. A corresponding point is searched for by the template in the image, and if the corresponding point candidate in the obtained reference image matches the center point of the original template, the corresponding point candidate in the searched image is found in the reference image. There is a method of judging that a point is a corresponding point, and if they do not match, it is not a corresponding point.

However, the concept of this method alone is not enough for FIG.
The corresponding point candidate of the J point in the image of FIG. 2 (a) is erroneously extracted as the K point in the image of FIG. 12 (b) by the background window. 12 is incorrectly extracted again as the point J in the image of FIG. 12A, that is, when the erroneous extraction is performed twice,
Although the corresponding point of the L point in the image of FIG. 12A is correctly extracted as the M point in the image of FIG. 12B, the corresponding point of the M point in the image of FIG. In many cases, erroneous determinations are made, for example, when erroneous extraction is performed due to N points and a background tree or ground.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems of the prior art, and a first object of the present invention is to provide a method that uses the entire area without discrimination between a subject and a background in an image to be used. An object of the present invention is to provide a corresponding point extraction method and apparatus capable of performing highly accurate corresponding point extraction processing.

A second object of the present invention is to perform high-accuracy corresponding point extraction processing over the entire area without discrimination between a subject and a background in an image to be used, and to shorten the processing time. And a corresponding point extraction method and apparatus.

A third object of the present invention is to accurately determine an occlusion region and a region where a distance greatly changes in an image to be used, and to perform a highly accurate corresponding point extraction process. It is an object of the present invention to provide a corresponding point extraction method and apparatus that can be used.

[0025]

According to a first aspect of the present invention, there is provided a corresponding point extracting method for determining correspondence between a plurality of images based on a template matching method. The template matching method is characterized in that two or more types of templates are used to determine corresponding points at points satisfying a specific condition.

According to a second aspect of the present invention, there is provided a method for extracting corresponding points according to the first aspect.
The template is characterized in that the area of the template is changed.

According to a third aspect of the present invention, there is provided a method for extracting a corresponding point according to the first aspect, wherein the two or more types of templates include:
The present invention is characterized by including a template obtained by deforming a template by an edge in the template.

According to a fourth aspect of the present invention, there is provided a method for extracting corresponding points according to the first aspect of the present invention, wherein the operation of the second and subsequent templates is performed by using the second and subsequent templates. Are performed on points that have become candidates in the calculation using the template of the above type.

According to a fifth aspect of the present invention, there is provided a method for extracting a corresponding point according to the first aspect of the present invention, comprising the steps of: It is characterized in that a calculation result based on a previous template is determined and then a calculation based on the next template is performed so as to use a template.

According to a sixth aspect of the present invention, there is provided a corresponding point extracting method according to the first aspect of the present invention, wherein two or more types of templates are simultaneously operated. Is selected from the calculation results.

According to a seventh aspect of the present invention, there is provided a method for extracting corresponding points according to the first aspect of the present invention, wherein the point satisfying the specific condition is a plurality of images. The distribution of parallax between the points is a point in a region where the distribution changes sharply.

According to another aspect of the present invention, there is provided a method for extracting corresponding points, wherein the distribution of parallax between the plurality of images changes sharply. It is characterized in that the determination as to whether or not the point is within the region is performed using the evaluation value of the template matching.

According to a ninth aspect of the present invention, in order to achieve the third object, in the corresponding point extracting method according to the seventh aspect, the distribution of parallax between the plurality of images changes sharply. The determination as to whether or not the point is within the region is performed using the evaluation value of the template matching and the determination result as to whether or not the reference point and the corresponding point of the point match the candidate. Things.

According to a tenth aspect of the present invention, in order to achieve the third object, in the corresponding point extracting method according to the seventh aspect, the distribution of parallax between the plurality of images changes sharply. The determination as to whether or not the point is within the region is performed by using the template included in the previous template at the point within the region where the disparity distribution of the parallax between the plurality of images changes steeply, and using each of the templates in the template as the similarity evaluation function. The difference between the pixel value of the element and the pixel value of the pixel in the image to be searched is obtained by using the number of pixels equal to or less than a certain threshold value.

In order to achieve the first object, a corresponding point extracting apparatus according to claim 11, wherein the correspondence between a plurality of images is clarified based on a template matching method. The method is characterized in that at points satisfying a specific condition, corresponding points are determined using two or more types of templates.

According to a twelfth aspect of the present invention, there is provided a corresponding point extracting apparatus according to the eleventh aspect, wherein the two or more types of templates change the area of the template. It is characterized in that it is a template that has been made.

According to a thirteenth aspect of the present invention, there is provided a corresponding point extracting apparatus according to the thirteenth aspect, wherein the two or more types of templates are defined by edges in the template. It is characterized by including a template obtained by deforming the template.

In order to achieve the second object, a corresponding point extracting apparatus according to claim 14 is the same as the corresponding point extracting apparatus according to claim 11, wherein the calculation using the second and subsequent templates is performed earlier. Are performed on points that have become candidates in the calculation using the template of the above type.

According to another aspect of the present invention, there is provided a corresponding point extracting apparatus according to the present invention, wherein two types of corresponding point extracting apparatuses are provided based on the calculation results of the first and subsequent templates. It is characterized in that a calculation result based on a previous template is determined and then a calculation based on the next template is performed so as to use a template.

In order to achieve the second object, a corresponding point extracting apparatus according to claim 16 performs the operation using two or more types of templates simultaneously in the corresponding point extracting apparatus according to claim 11, wherein Is selected from the calculation results.

According to a seventeenth aspect of the present invention, in the corresponding point extracting apparatus according to the eleventh aspect, the point satisfying the specific condition is defined by a plurality of images. The distribution of parallax between the points is a point in a region where the distribution changes sharply.

In order to achieve the first object, a corresponding point extracting device according to claim 18 is the corresponding point extracting device according to claim 17, wherein the distribution of parallax between the plurality of images changes sharply. It is characterized in that the determination as to whether or not the point is within the region is performed using the evaluation value of the template matching.

According to a third aspect of the present invention, in order to achieve the third object, in the corresponding point extracting apparatus according to the seventeenth aspect, the distribution of parallax between the plurality of images changes sharply. The determination as to whether or not the point is within the region is performed using the evaluation value of the template matching and the determination result as to whether or not the reference point and the corresponding point of the point match the candidate. Things.

According to a twentieth aspect of the present invention, in order to achieve the third object, in the corresponding point extracting apparatus according to the seventeenth aspect, the distribution of parallax between the plurality of images changes sharply. The determination as to whether or not the point is within the region is performed by using the template included in the previous template at the point within the region where the disparity distribution of the parallax between the plurality of images changes steeply, and using each of the templates in the template as the similarity evaluation function. The difference between the pixel value of the element and the pixel value of the pixel in the image to be searched is obtained by using the number of pixels equal to or less than a certain threshold value.

In order to achieve the first object, a corresponding point extracting method according to a twenty-first aspect of the present invention provides a method for extracting a template from a single image among a plurality of input images based on a predetermined criterion first. Is created, the similarity is determined using the pixel value of each pixel of the created template and the pixel value of the pixel in the input other image, and a specific condition is determined from the obtained similarity. Another template is created based on the determination result.

In order to achieve the first object, a corresponding point extracting method according to a twenty-second aspect is the same as the corresponding point extracting method according to the twenty-first aspect, wherein the template is created by changing the area of the template. It is characterized by the following.

In order to achieve the first object, a corresponding point extracting method according to a twenty-third aspect is characterized in that, in the corresponding point extracting method according to the twenty-first aspect, the determination of a specific condition is based on the input similarity. It is characterized in that the comparison is performed with a certain threshold value.

In order to achieve the first object, a corresponding point extracting method according to a twenty-fourth aspect is characterized in that, in the corresponding point extracting method according to the twenty-first aspect, the determination of a specific condition is based on a plurality of inputted similarities. It is characterized in that the judgment is made when the difference between the degrees is smaller than a certain threshold value.

In order to achieve the first object, a corresponding point extracting method according to a twenty-fifth aspect is characterized in that, in the corresponding point extracting method according to the twenty-first aspect, the determination of a specific condition is based on parallax between a plurality of images. The method is characterized by determining pixels in an area where the distribution changes sharply.

According to a twenty-sixth aspect of the present invention, there is provided a method for extracting a corresponding point, comprising the steps of: Is the template size of the point of the pixel near the center point of the template.

According to a twenty-seventh aspect of the present invention, in the corresponding point extracting method according to the twenty-first aspect, the predetermined criterion for creating an initial template is provided. Is characterized by the presence or absence of a strong edge in the template.

In order to achieve the first object, the corresponding point extracting apparatus according to the twenty-eighth aspect is characterized in that a template is first selected from one of a plurality of input images based on the predetermined criterion. And a similarity calculation for calculating a similarity using a pixel value of each pixel of the template created by the template creating unit and a pixel value of a pixel in the input other image. Means, a specific condition determining means for determining a specific condition from the similarity obtained by the similarity calculating means, and a second template for creating another template based on the determination result of the specific condition determining means.
And a template creation means.

In order to achieve the first object, a corresponding point extracting apparatus according to claim 29 is characterized in that in the corresponding point extracting apparatus according to claim 28, the second template creating means reduces the area of the template. It is characterized in that a changed template is created.

In order to achieve the first object, a corresponding point extracting apparatus according to claim 30 is characterized in that, in the corresponding point extracting apparatus according to claim 28, the specific condition judging means determines whether the inputted similarity is equal to the inputted similarity. It is characterized in that a specific condition is determined by comparing with a certain threshold value.

In order to achieve the first object, a corresponding point extracting apparatus according to claim 31 is the corresponding point extracting apparatus according to claim 28, wherein the specific condition judging means includes a plurality of inputted similarities. And determining a specific condition by determining when the difference is smaller than a certain threshold value.

According to a thirty-second aspect of the present invention, in the corresponding point extracting apparatus according to the thirty-second aspect, the specific condition judging means includes a disparity distribution between a plurality of images. Is characterized by judging a pixel in an area where abruptly changes, and judging a specific condition.

In order to achieve the second object, a corresponding point extracting apparatus according to claim 33 is the corresponding point extracting apparatus according to claim 28, wherein the predetermined criterion for creating an initial template is the same as that of the corresponding point extracting apparatus. Is the template size of the point of the pixel near the center point of the template.

According to a thirty-fourth aspect of the present invention, there is provided a corresponding point extracting apparatus according to the twenty-eighth aspect, wherein the predetermined criterion for creating an initial template is the same as that of the corresponding point extracting apparatus. Is characterized by the presence or absence of a strong edge in the template.

In order to achieve the second object, a corresponding point extracting method according to claim 35 is characterized in that a template size used at a point obtained immediately before from one of a plurality of input images is used. A template is created using the calculated template, and a similarity is calculated using the pixel values of the respective pixels of the created template and the pixel values of the pixels in the input other image. It is characterized in that a judgment is made when the similarity is smaller than a certain threshold value, and a template in which the area of the template is changed is created based on the judgment result.

In order to achieve the second object, a corresponding point extracting method according to claim 36 is characterized in that a template size used at a point obtained immediately before from one of a plurality of input images is used. A template is created using the calculated template, and a similarity is calculated using the pixel values of the respective pixels of the created template and the pixel values of the pixels in the input other image. It is characterized in that a judgment is made when a plurality of differences in similarity are smaller than a certain threshold value, and a template in which the area of the template is changed is created based on the judgment result.

In order to achieve the second object, a corresponding point extracting method according to claim 37 is characterized in that a template size used at a point obtained immediately before from one of a plurality of input images is used. A similarity is calculated using the pixel value of each pixel of the generated template and the pixel value of the pixel in the input other image, and a plurality of images are calculated from the calculated similarity. The method is characterized in that pixels in an area where the distribution of parallax changes sharply are determined, and a template in which the area of the template is changed is created based on the determination result.

In order to attain the second object, the corresponding point extracting apparatus according to claim 38 uses a template size used at a point obtained immediately before from one of a plurality of input images. A first template creation unit for creating a template using the first template creation unit, and a pixel value of each pixel of the template created by the first template creation unit and a pixel value of a pixel in the input other image. Similarity calculating means for obtaining similarity, determining means for determining when the similarity input from the similarity calculated by the similarity calculating means is smaller than a certain threshold value, and a template based on the determination result of the determining means And a second template creating means for creating a template having a changed area.

In order to attain the second object, the corresponding point extracting apparatus according to claim 39 uses a template size used at a point obtained immediately before from one of a plurality of input images. A first template creation unit for creating a template using the first template creation unit, and a pixel value of each pixel of the template created by the first template creation unit and a pixel value of a pixel in the input other image. Similarity calculating means for obtaining similarity, determining means for determining when a difference between a plurality of similarities input from the similarity calculated by the similarity calculating means is smaller than a certain threshold value, and determining by the determining means And a second template creating means for creating a template in which the area of the template is changed based on the result.

In order to achieve the second object, the corresponding point extracting apparatus according to claim 40, further comprises a template size used at a point obtained immediately before from one of a plurality of input images. A first template creation unit for creating a template using the first template creation unit, and a pixel value of each pixel of the template created by the first template creation unit and a pixel value of a pixel in the input other image. Similarity calculating means for obtaining similarity; determining means for determining, from the similarity calculated by the similarity calculating means, a pixel in an area in which the distribution of parallax between a plurality of images changes sharply; And a second template creating means for creating a template in which the area of the template is changed based on.

[0065]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below with reference to FIGS.

(First Embodiment) First, the first embodiment of the present invention will be described.
The embodiment will be described with reference to FIGS. FIG.
FIG. 2 is a block diagram showing a configuration of a corresponding point extraction device according to the first embodiment of the present invention. In the figure, reference numeral 101 denotes image input means, for example, two cameras as shown in FIG. Have. 102 is a reference image memory, 103 is an image memory to be searched, 104 is an address generation unit, 105 is a data storage unit, 106 is a template image creation unit, 107
Denotes an evaluation function calculation unit, 108 denotes a determination unit, and 109 denotes a template size change unit.

In FIG. 1, an image is input from the image input means 101 and the input image is stored in the reference image memory 1.
02 and the image memory 103 to be searched for, respectively. From the address generation unit 104, the reference image memory 1
02 is an address indicating one point in the data storage unit 1
From 05, the initial template size is sent to the template image creation unit 106, and the template image creation unit 106 creates a template image of the template size around the point indicated by the address using the reference image memory 102. You. In the evaluation function calculation unit 107, the expressions (1) to (3) described in the related art are used between the template image input from the template image creation unit 106 and the searched image input from the searched image memory 103. The evaluation value is calculated by using one of the similarity evaluation functions in the expression, and the corresponding point candidate and the evaluation value of the point are output. The determination unit 108 determines whether or not the center point of the template is a point in an area where the distance greatly changes in the image from the obtained evaluation value if the template is a large template. Change part 1
09, a new template size is sent to the template image creation unit 106, and the evaluation function is calculated again by the evaluation function calculation unit 107 using the new template created there, and the corresponding points are determined. If the center point of the template is not a point in the area where the distance greatly changes in the image, the corresponding point result is output.

In the above configuration, the address generator 10
4 executes an operation of generating an address indicating one point in the reference image memory 102, the data storage unit 105 stores template size data for the first template matching, and stores the data in the template image generation unit 1
The operation of sending to 06 is performed. Template image creation unit 1
Reference numeral 06 receives the address from the address generation unit 104, the template size data from the data storage unit 105 or the template size from the template size change unit 109, and the reference image from the reference image memory 102, and executes the operation of creating the template image. Evaluation function operation unit 1
In step 07, the evaluation value of the template image and the searched image is calculated based on one of the above equations (1) to (3), and the operation of outputting the corresponding point candidate and its evaluation value is executed.

The processing operation of the evaluation function calculation unit 107 will be described with reference to the flowchart of FIG. In this flowchart, the evaluation function uses the correlation value σ of the above equation (2).
(X, y) is used.

It is assumed that the coordinate value of the corresponding point candidate is (X, Y) and the correlation value is σ (x, y).

In step S201 of FIG. 2, a template image and an image to be searched are input, and an evaluation value is initialized in the next step S202 (here, σx, y =
0). Next, one point is selected from the image searched in step S203, the point is aligned with the center of the template, and an evaluation value is calculated in the next step S204. Next, step S
In step 205, it is determined whether or not the evaluation value calculated in step S204 is higher than the evaluation value up to now (because it is a correlation value). If it is determined to be higher, the coordinates are determined as corresponding point candidates in next step S206. After the previous evaluation values are replaced, the process proceeds to the next step S207. If it is determined in step S205 that the calculated evaluation value is lower than the previous evaluation value, the process skips step S206 and proceeds to step S207.

In this step S207, it is determined whether or not processing has been performed for all points in the image to be searched. If it is determined that processing has been performed for all points, the processing is held in the next step S208. After outputting the coordinate values (X, Y) and the evaluation values (here, σx, y) of the corresponding corresponding point candidates, this processing operation ends. Step S2
If it is determined in step 07 that processing has not been performed for all points, the process returns to step S203 to obtain an evaluation value for the next point, selects the next one point from the searched image again, and evaluates again. Performs a value operation.

As described in detail above, the evaluation function calculator 107
Works.

The determination unit 108 determines that the template size is 1
If the size is the first time, an operation of determining whether or not the center point of the template is a point in an area where the distance greatly changes is executed based on the obtained evaluation value. If the template size is 2
If the size is the first time, it operates to output the corresponding point candidate as the corresponding point. The determination unit 108 determines whether or not the center point of the template is a point in a region where the distance greatly changes by determining whether or not the evaluation value is higher than a certain threshold value (here, σth). is there. because,
When the template 31 is created around the point P in FIG. 3A and the corresponding point is obtained in FIG. 3B, for example, when the evaluation value at a point on the line L is plotted, FIG. become that way. Here, as for the evaluation value, the point P using σ in the above equation (2) is not a point in the region where the distance greatly changes, and a peak having a high evaluation value σ appears as shown in FIG. The corresponding points can be determined.

However, when the template 32 is created centering on the point Q in FIG. 3A and the corresponding points are obtained in FIG. 3B, for example, when the evaluation value at a point on the line M is plotted, The result is as shown in FIG. This is because, as shown in the above-mentioned problem, there are regions in the template having greatly different distances, and as shown in FIG. 3C, the mountain whose evaluation value is improved by the region R in the template is the mountain on the left side. , Area S
The mountain whose evaluation value is improved is the mountain on the right. As described above, the evaluation value of the corresponding point candidate of the point existing in the area where the distance greatly changes is considerably lower than the evaluation value of the point in the other area (when σ is used). Due to such a fact, it is possible to determine whether or not the center point of the template is a point in an area where the evaluation value is higher or lower than a certain threshold value σth. This judgment is based on the evaluation value Ε (x, y) if the above equation (1) is used for calculating the evaluation value.
Is higher than the threshold value σth, the point is in the area. If the above equation (2) is used, if the evaluation value σ (x, y) is lower than the threshold value σth, the point is in the area. Further, if the above equation (3) is used, if the evaluation value C (x, y) is lower than the threshold value Cth, the point is within the area.

When the determining unit 108 determines that the center point of the template is in a region where the distance changes greatly, the template size changing unit 109 changes the template size to a smaller size and outputs it to the template image creating unit 106. Works.

When the components operating as described in detail above are configured as shown in FIG. 1, highly accurate corresponding point extraction processing can be performed, and a highly accurate object distance distribution can be obtained.

In the present embodiment, there are two types of template sizes, large and small. However, if the corresponding points are still difficult to understand even if the template size is small, templates of various sizes are used. You may.

(Second Embodiment) Next, a second embodiment of the present invention will be described.
Will be described with reference to FIGS.

FIG. 5 is a block diagram showing a configuration of a corresponding point extracting apparatus according to a second embodiment of the present invention. In FIG. 5, the same parts as those in FIG. The code is attached. 5 is different from FIG. 1 in that the template size changing unit 109 differs from the configuration in FIG.
Is deleted, and the edge mask template creating unit 110 is added.

In FIG. 5, as in the first embodiment,
An image is input from the image input unit 101, and the input image is stored in the reference image memory 102 and the searched image memory 103, respectively. Address generator 104
, An address indicating one point in the reference image memory 102 is sent from the data storage unit 105, and an initial template size is sent to the template image creation unit 106 from the data storage unit 105.
The template image creating unit 106 creates a template image of the template size around the point indicated by the address using the reference image memory 102. In the evaluation function calculation unit 107, the expressions (1) to (3) described in the related art are used between the template image input from the template image creation unit 106 and the searched image input from the searched image memory 103. The evaluation value is calculated by using one of the similarity evaluation functions in the expression, and the corresponding point candidate and the evaluation value of the point are output. If the template is a large template, the determining unit 108 determines from the obtained evaluation value whether or not the center point of the template is a point in an area where the distance greatly changes in the image. A signal is sent to the edge mask template creating unit 110, the template image is changed so as to mask the outside of the strong edge, the evaluation function is calculated again by the evaluation function operation unit 107 using the new template image, and the corresponding point is determined. decide. If the center point of the template is not a point in the area where the distance greatly changes in the image, the corresponding point result is output.

The above configuration differs from the first embodiment in an edge mask template creating section 110. In the first embodiment, when the center point of the template is an inner point in the image, the template size is reduced. However, in the present embodiment, the template is divided by strong edges and matching is performed using the template including the center point. I do. This is because the texture is different in a region where the distance greatly changes, and a strong edge appears. Therefore, dividing the template eliminates the influence of the evaluation values of the portions having different distances in the template.

Next, an edge mask template creating section 11
The processing operation of 0 will be described with reference to the flowchart of FIG. In the present embodiment, a template is masked so as not to be used for the evaluation value calculation, instead of dividing the template.

In step S601 of FIG. 6, a template image is input, and in the next step S602, an edge image of the template image is created by a technique such as a Sobel filter. Next, an edge stronger than the threshold value in step S603 is extracted from the edge created in step S602. Next, in step S604, the step S
In step 603, an object close to the template center point is extracted from the strong edge images obtained. Next, step S605
2. Mask the outside by dividing into the inside and outside of the edge as viewed from the center of the template. Next, in step S606, the mask image masked in step S605 and the template image are overlapped to create an edge mask template image, and then this processing operation ends.

FIGS. 7A and 7B schematically show the above processing. FIG. 7A shows a template image, FIG. 7B shows an edge image, FIG. 7C shows a strong edge image, FIG.
(E) is a mask image, and (f) is an edge mask template image.

[0086] By performing the corresponding point extraction using the edge mask template image, it takes extra processing time, but it is possible to perform the corresponding point extraction with high accuracy and obtain the distance distribution of the object with high accuracy. Becomes

In this embodiment, an edge close to the template is selected. However, since the edge is not limited to an edge separating the subject from the background, an edge mask template image is created sequentially from an edge far from the center. It is effective to repeat it many times.

(Third Embodiment) Next, a third embodiment of the present invention will be described.
The embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing the processing operation of the evaluation function calculation unit in the corresponding point extraction device according to the third embodiment of the present invention.

In the first and second embodiments described above,
When the center point of the template is a point in a region where the distance greatly changes, the template size is reduced or template matching is performed again on the edge mask template image for the entire region in the searched image. However, performing template matching with a small template size or an edge mask template image is not only a cause for other similar points, causing an erroneous correspondence but also requires a long processing time.

Therefore, in the present embodiment, the second and subsequent evaluation value calculations are performed only at points where the first evaluation value is higher than a certain threshold value by using the result of the template matching performed first. It is.

In the flowchart of FIG. 8, the correlation value σ (x, y) of the above equation (2) is used for the evaluation value calculation, and the coordinate value of the corresponding point candidate is (X, Y). The value is σ (X,
Y) The array holding the first evaluation value is H (x, y), and after changing the template size, the threshold value used to determine whether to perform the evaluation value calculation on the edge mask template image is Hth And

In step S801 in FIG. 8, a template image and an image to be searched are input, and an evaluation value is initialized in the next step S802 (here, σx, y =
0). Next, one point is selected from the searched image in step S803, and in the next step S804, it is determined whether or not the evaluation function calculation of this point is the first time. If it is the first time, the process proceeds to the next step S805. If it is not the first time, it is determined in step S811 whether the evaluation value H (x, y) at that point is higher than the threshold value Hth (good). Judgment (in this case, the correlation value is larger, the larger is better). If it is lower (bad), the process returns to step S803 to select the next one point from the image to be searched. S8
Go to 05.

In this step S805, the evaluation value is calculated, and in the next step S806, the evaluation value (here, σ (x, y)) calculated in step S805 is written in H (x, y). Next, in step S807, the aforementioned step S805 is performed.
It is determined whether or not the evaluation value calculated in is higher than the previous evaluation value. If it is determined that the evaluation value is higher, the next step S
At step 808, the coordinates are held as the corresponding point candidates, and the evaluation values (σX, Y in this case) are replaced, and the next step S
Go to 809. If it is determined that the evaluation value calculated in step S807 is lower than the previous evaluation value, step S808 is skipped and step S808 is skipped.
Go to 809.

In this step S809, it is determined whether or not processing has been performed for all points in the image to be searched. If it is determined that processing has been performed for all points, the next step S810 holds the data. After outputting the coordinate values (X, Y) and the evaluation values (here, σx, y) of the corresponding corresponding point candidates, this processing operation ends. Step S8
If it is determined in step 09 that all points have not been processed, the process returns to step S803 to obtain an evaluation value for the next point, selects the next one point from the searched image again, and evaluates again. Performs a value operation.

As described in detail above, since it is determined whether or not the evaluation value calculation must be performed using the previous evaluation value when the template size is different or the like, the evaluation value calculation is performed. Compared with the first and second embodiments, the corresponding point extraction processing time is shortened, the possibility of erroneous extraction is reduced, and the corresponding point extraction accuracy is improved. Then, a highly accurate distance distribution of the subject can be obtained.

(Fourth Embodiment) Next, a fourth embodiment of the present invention will be described.
The embodiment will be described with reference to FIGS.

In the above-described first to third embodiments, when it is determined that the center point of the template is a point in an area where the distance changes greatly, the template size is changed and the evaluation function is calculated again. I was going. However, in this method, the template moves a plurality of times over the entire screen to be searched, and the processing time becomes longer.

Therefore, in the present embodiment, as shown in FIG. 11, the template is divided into template A and template B, and at the same time, the evaluation value in each area is obtained, and the evaluation value in a certain area is used. It is designed to determine which evaluation value to use.

FIG. 9 is a block diagram showing the configuration of a corresponding point extracting apparatus according to this embodiment. In FIG. 9, the same parts as those in FIG. 1 of the first embodiment are denoted by the same reference numerals.
9 differs from FIG. 1 in that the template size changing unit 109 is removed from the configuration in FIG.

In FIG. 9, an image is input from the image input means 101 and the input image is stored in the reference image memory 1.
02 and the image memory 103 to be searched for, respectively. From the address generation unit 104, the reference image memory 1
02 is an address indicating one point in the data storage unit 1
05, the template size is sent to the template image creation unit 106, and the template image creation unit 10
In 6, a template image of the template size is created around the point indicated by the address using the reference image memory 102. In the evaluation function calculation unit 107, between the template image input from the template image creation unit 106 and the searched image input from the searched image memory 103, the expressions (1) to (3) are used. The evaluation value is calculated using the evaluation function of one similarity, and the corresponding point candidate and the evaluation value of the point are output. At the same time, a small template size is input from the data storage unit 105, an evaluation value within that range is calculated, and a corresponding point candidate and its evaluation value are output. The determination unit 108 determines from the evaluation value of the template A whether the center point of the template is a point in an area where the distance greatly changes in the image, and if the point is in the area, the corresponding point obtained by the template B Output candidates as corresponding points. If the center point of the template is not a point in the region where the distance greatly changes in the image, the corresponding point candidate obtained for template A is output as the corresponding point.

FIG. 10 is a flowchart showing the processing operation of the evaluation function calculation unit 107 in the corresponding point extracting apparatus according to the present embodiment.

In this flowchart, the correlation value σ (x, y) of the above equation (2) is used for the evaluation value calculation, the coordinate value of the corresponding point candidate of the template A is (X1, Y1), and the evaluation of the point is performed. The value is σ1 (X, Y), the coordinate value of the corresponding point candidate of template B is (X2, Y2), the evaluation value of that point is σ2 (X, Y), and the evaluation value of template A is good or bad. The threshold value used to determine whether or not the threshold value is Hth.

In step S1001 of FIG. 10, a template image and an image to be searched are input, and an evaluation value is initialized in the next step S1002 (σ1X, Y = σ2X, Y
0). Next, 1 is selected from the image searched in step S1003.
A point is selected, and in the next step S1004, the template A
And the evaluation value of the template B are calculated at the same time.

The evaluation value of template A is obtained by the following equation (4), and the evaluation value of template B is obtained by the following equation (5).

[0105]

(Equation 4)

[0106]

(Equation 5) At this time, as can be seen from FIG. 11, the template B is smaller than the template A and is included in the area of the template A. Therefore, when the calculation for obtaining the evaluation value σ1 (x, y) of the template A is performed, the evaluation value of the template B is simultaneously obtained. σ2 (x, y)
Can also be requested.

In the next step S1005, the step S1005 is executed.
It is determined whether the evaluation value of template A (here, σ1 (x, y)) calculated in 1004 is higher than the previous evaluation value (σ1X, Y), and if it is determined to be higher, the next step After holding the coordinates as the corresponding point candidate of the template A in S1006 and replacing the previous evaluation values, and when it is determined that the evaluation value calculated in step S1005 is lower than the previous evaluation value, Step S1
Skip 006 and proceed to step S1007.

In this step S1007, the evaluation value of template A calculated in step S1004 is higher than threshold value Hth and the evaluation value of template B (σ
2 (x, y)) is determined to be higher than the previous evaluation value.
If the evaluation value of the template A is higher than the threshold value Hth and the evaluation value (σ2 (x, y)) of the template B is higher than the previous evaluation value, the corresponding point candidate of the template B is determined in the next step S1008. After the coordinates are held and the evaluation values thus far are replaced, the process proceeds to the next step S1009.
In step S1007, the template A
Is not higher than the threshold value Hth and the evaluation value (.sigma.2 (x, y)) of the template B is not lower than the previous evaluation value, the step S1008 is skipped and the step S1009 is skipped. Proceed to.

In step S1009, it is determined whether or not processing has been performed for all points in the image to be searched. If it is determined that processing has been performed for all points, the processing is held in step S1010. (X1, Y1), (X2, Y2) and evaluation value (σ1
After output of (x, y), this processing operation ends. If it is determined in step S1009 that all points have not been processed, the process returns to step S1003 to select the next one point from the image searched again to obtain an evaluation value for the next point. Then, the evaluation value is calculated again.

Next, in the judging section 108, the evaluation value of the template A calculated by the evaluation value function calculating section 107 (σ1
x, y) to determine whether or not the center point of the template is a point in a region where the distance greatly changes, and if the point is in that region, the coordinate value of the corresponding point candidate obtained in template B (X2, Y2) is output as the corresponding point, and if it is not a point in the area, the coordinate value (X1, Y1) of the corresponding point candidate obtained by template A is output as the corresponding point.

As described in detail above, a small area is provided in one template, and the evaluation value calculation in that area is performed in the same manner. When the center point of the template is a point in the area where the distance greatly changes. By using the evaluation value calculation result of the small area in the template, the corresponding point extraction accuracy is improved with almost no increase in the corresponding point extraction processing time as compared with the conventional example. Then, a highly accurate distance distribution of the subject can be obtained.

In the present embodiment, one small area is provided in the template. However, the number of the small areas may be any number, and an edge mask is applied as in the second embodiment. Similar effects can be obtained by calculating the evaluation values simultaneously.

(Fifth Embodiment) Next, a fifth embodiment of the present invention will be described.
The embodiment will be described with reference to FIGS. For example, in an image obtained by means shown in FIGS. 24A and 24B, there is a problem that an area called an occlusion area having no corresponding point exists, and it is difficult to detect the area. .

For example, the point corresponding to the point I in FIG. 14A is hidden by the car as the subject in FIG. 14B, and does not exist in this image. Such an area is called an occlusion area. Since there is no corresponding point at a point in this area, the result of a point in this area must not be used as a corresponding point extraction result. As a method for detecting this region, if a center point candidate in the reference image is obtained in the image to be searched, then a template is created in the image searched around that point, and the template is used in the reference image. In the template, the corresponding point is searched for, and if the corresponding point candidate in the obtained reference image matches the center point of the original template, the corresponding point candidate in the searched image is the point of the point in the reference image. There is a method of judging that a point is a corresponding point, and if they do not match, it is not a corresponding point.
However, with this concept alone, the corresponding point candidate at point J in FIG.
When the corresponding point in the template centered on the point K is erroneously extracted again as the point J in FIG. 14A, that is, when the erroneous extraction is performed twice, 14), the corresponding point of L point is correctly extracted as the M point in FIG. 14B, but the corresponding point of the M point is N point in FIG. In many cases, erroneous judgments are made when erroneous extraction is performed.

In order to solve such a problem, as shown in FIG. 15A, a template 151 having the same size as that used in another region is used at a point D in a region where the distance greatly changes. Using a template 152 that is relatively small, or a region 15 divided by an edge as shown in FIG.
Only three need be used for the calculation.

In the present embodiment, such an occlusion area and an area in which the distance greatly changes are determined, and if a corresponding point candidate is obtained in the searched image shown in the above-mentioned problem. A template based on the corresponding point candidate is created, a corresponding point candidate is determined in the reference image, and a determination based on the evaluation value is performed by a method of determining whether the point matches the point in the reference image. In addition, a region where the distance changes greatly and an occlusion region are determined.

In this embodiment, the processing contents of the evaluation function calculator and the judging unit are different from those of the first to fourth embodiments.

FIG. 12 is a flowchart showing the processing operation of the evaluation value function calculating section in the corresponding point extracting device according to the present embodiment. In this flowchart, the evaluation value calculation uses the correlation value σ (x, y) of the above equation (2), and the coordinate value of the corresponding point candidate in the searched image is (XL, YL). The evaluation value is σL (X, Y), the coordinate value of the corresponding point candidate of the corresponding point candidate in the reference image is (XR, YR), and the evaluation value of the point is σ
Let R (X, Y).

In step S1201 of FIG. 12, a template image and an image to be searched are input, and an evaluation value is initialized in the next step S1202 (σRX, Y = σLX, Y
= 0). Next, one point is selected from the images searched in step S1203, and an evaluation value is calculated in the next step S1204. In the next step S1205, step S120
It is determined whether the evaluation value (σL (x, y)) calculated in step 4 is higher than the evaluation value (σL (1X, Y)) up to now, and if it is determined to be higher, the next step S1206 is performed. After holding the coordinates as the corresponding point candidate and replacing the evaluation value σL (1X, Y) so far, the process proceeds to step S1207. If it is determined in step S1205 that the calculated evaluation value is lower than the previous evaluation value, the process proceeds to step S1206.
Skip to step S1207.

In step S1207, it is determined whether or not processing has been performed for all points in the searched image.
If it is determined that the processing has not been performed for all points, the process returns to step S1203 to select the next one point from the searched image and perform the evaluation value calculation again. If it is determined in step S1207 that processing has been performed for all points, a template is created from the image searched for in the next step S1208, centering on the coordinates of the corresponding point candidate. Next, one point is selected from the reference image in step S1209, and an evaluation value is calculated in the next step S1210. Next, in step S1211, the above-described step S1 is executed.
It is determined whether or not the evaluation value (σR (x, y)) calculated in 210 is higher than the evaluation value (σR (X, Y)) up to now. After the coordinates are held as corresponding point candidates and the evaluation values (σR (X, Y)) so far are replaced, the process proceeds to step S1213. If it is determined in step S1211 that the evaluation value (σR (x, y)) is not higher than the previous evaluation value (σR (X, Y)), step S1212 is skipped and step S1212 is skipped.
Proceed to 1213. .

In this step S1213, it is determined whether or not processing has been performed for all points in the searched image.
If it is determined that the processing has been performed for all points, in the next step S1214, the coordinate values (XL, YL) of the corresponding point candidates held at that time and the evaluation values (σL (X, Y))
And the coordinate value (XR, YR) of the corresponding point candidate and its evaluation value (σ
R (x, y)), the processing operation ends. Also,
If it is determined in step S1213 that all points have not been processed, the process returns to step S1209 to obtain an evaluation value for the next point, and selects the next one point from the searched image again. The evaluation value is calculated again.

Next, in the judging unit 108, the coordinate value (XR, YR) of the corresponding point candidate of the corresponding point candidate matches the center point of the template created from the reference image, and the evaluation values are respectively equal to the evaluation values. If it is higher than the threshold value (in this case, if the threshold value is σth, σL (X, Y)> σHth and σR
If (X, Y)> σHth), the corresponding point candidate is determined to be a corresponding point and output. Also, the coordinate value (XR, YR) of the center point of the template created from the reference image and the corresponding point candidate of the corresponding point candidate does not match, and even if one of the evaluation values is smaller than the second threshold value of the evaluation value If low (in this case, if the threshold is σLth, σL (X, Y) <σHth or σR (X, Y) <σ
If Hth), the center point of the template is determined to be a point in the occlusion area, and no corresponding point exists.
In other cases, the center point of the template is determined to be a point in the area where the distance changes greatly.If the template size is large, the template size is changed and the evaluation value calculation is performed again. It is determined as an unknown point.

FIG. 13 is a table summarizing the above concept. FIG. 9A shows a case where the template size is large and the coordinate value (XR, YR) of the corresponding point candidate of the corresponding point candidate matches the center point of the template created from the reference image.
FIG. 6B shows a case where the template size is large and the coordinate value (XR, YR) of the corresponding point candidate of the corresponding point candidate does not match the center point of the template created from the reference image, and FIG. FIG. 11D shows a case where the coordinate value (XR, YR) of the corresponding point candidate of the corresponding point candidate coincides with the center point of the template created from the reference image with a small size.
Indicates a case where the template size is small and the coordinate value (XR, YR) of the corresponding point candidate of the corresponding point candidate does not match the center point of the template created from the reference image.

In FIG. 13, “pair” indicates that the corresponding point is set, “o” indicates that the occlusion area is set, “small” indicates that the template size is reduced and the determination process is performed again, and “ "Not" indicates that the corresponding point is unknown.

As described in detail above, not only the evaluation value E but also the corresponding point candidate of the corresponding point candidate is used in the determination of the area and the occlusion area where the distance greatly changes, so that the corresponding point by the point in the occlusion area is used. And the corresponding point extraction accuracy is improved. In addition, the reliability of the corresponding point candidate is improved at points in other areas.

In the present embodiment, FIG.
As shown in (d), there is a point that is unknown when it is obtained with a small template size. At this time, other means such as further reducing the template size or using an edge mask template image may be used. .

In addition to the regions where the distance greatly changes as described in the first to fifth embodiments, even when a moving subject is imaged, the distribution of the parallax is sharp due to the motion parallax. Even in the case of a change, a similar effect can be obtained by applying these embodiments. Further, as an example of the evaluation function for obtaining the similarity, the correlation value of the equation (2) shown in the conventional example is used, but the equations (1) to (3) are used.
The same effect can be obtained by using any of the equations. However, according to the experiment of the present inventors, the corresponding point extraction accuracy was highest when the number of pixels equal to or less than a certain threshold value of the equation (3) was used.

(Sixth Embodiment) First, the sixth embodiment of the present invention will be described.
The embodiment will be described with reference to FIGS. FIG.
FIG. 6 is a block diagram showing a configuration of a corresponding point extracting apparatus according to a sixth embodiment of the present invention. In FIG. 6, the same reference numerals as those in FIG. 1 of the first embodiment denote the same parts. It is attached. 16 is different from FIG. 1 in that the evaluation function operation unit 107 is deleted from the configuration of FIG.
11 is added.

In FIG. 16, for example, an image is input from image input means having two cameras as shown in FIG. 24 (a), and the input image is stored in the reference image memory 102.
And stored in the image memory 103 to be searched. From the address generation unit 104, the reference image memory 10
The address indicating one of the two points is sent to the data storage unit 105, and the template size is determined with reference to the template sizes of points around the point. The determined template size is sent from the data storage unit 105 to the template image creating unit 106, and the template image creating unit 106 uses the reference image memory 102 to center the template at the point indicated by the address. An image is created. In the pixel number calculation unit 111, a description has been given of the conventional technique between the template image input from the template image creation unit 106 and the searched image input from the searched image memory 103 (3).
Based on the formula, the number of pixels of a difference between each pixel in the template and one pixel on the searched image in the entire template image is calculated below a certain threshold ε, and the corresponding point candidate and the similarity of the point are calculated. Is output. When the calculation is performed for the first time, the determination unit 108 determines whether the template has a suitable size.
09, a new template size data changed based on the signal is sent to the data storage unit 105, and a new template size is sent from the data storage unit 105 to the template image creation unit 106, where it is created. Pixel number calculation unit 1 using new template
At 11, the similarity is calculated again, and the corresponding point is determined. Also,
If the calculation is for the second time, the corresponding point extraction result is output.

In the above configuration, the address generator 10
4 executes an operation for generating an address indicating one point in the reference image memory 102, and the data storage unit 105 performs the operation based on the template size used in the corresponding point extraction processing around the point indicated by the obtained address. An operation of determining the template size at the point and transmitting the determined template size to the template image creating unit 106 is performed.

The processing operation of the data storage unit 105 will be described with reference to the flowchart of FIG. Here, it is assumed that the template size to be used is St, and the St is set to k as an initial value (St =
k: k is a fixed value). The template size sent from the template size changing unit 109 is Sn,
It is assumed that Sn is determined by the template size changing unit 109.

In step S1701 of FIG. 17, it is determined whether to determine the template size used for the first processing or the template size used for the second processing. When the template size to be used for the first processing is obtained, in the next step S1702, the template sizes of eight points around the point indicated by the address from the address generation unit 104 are checked, and the smallest template size is determined. Determined as Ss. Next, step S170
In step 3, the small template size Ss is compared with the template size St of the point among the template sizes of the eight surrounding templates, and if St is larger than Ss, the template of the small eight points in the surrounding neighborhood is determined in the next step S1704. Let the size be the template size for that point (St =
Ss) Then, the process proceeds to step S1705.

In step S1701, 2
If the template size for the second processing is to be obtained, the template size St at that point is changed to the template size Sn from the template size changing unit 109 in step S1706 (St = Sn), and then step S17
Go to 05. In this step S1705, the template size St is sent to the template image creation unit 106.

As described above, by determining the template size with reference to the template size of another point,
An optimum template size is obtained, and the corresponding point extraction processing speed is improved.

Thereafter, the template image creation unit 106 sends the address and data storage unit 10 from the address generation unit 104.
5 and the reference image from the reference image memory 102, and executes the operation of creating the template image. The pixel number calculation unit 111 determines the similarity between the template image and the image to be searched based on the above equation (3). Then, the corresponding point candidate and the similarity are output.

FIG. 18 is a flowchart showing the processing operation of the pixel number calculation unit 111. In this flowchart,
The coordinate value of the corresponding point candidate is (X, Y), and each point (x, y)
Let C (x, y) be the similarity calculated by the above equation (3), Cxy be the variable that stores the maximum similarity, and Cxy2 be the variable that stores the second similarity.

In step S1801 of FIG. 18, a template image and an image to be searched are input, and variables Cxy and Cxy2 are initialized in the next step S1802 (Cxy = 0, Cxy2 = 0). Next, step S180
In step S18, one point (x, y) is selected from the image searched in step 3, and the point is aligned with the center of the template.
At step 04, the similarity C (x, y) is calculated. Next, step S
In 1805, it is determined whether or not the similarity C (x, y) calculated in step S1804 is greater than the variable Cxy. If it is determined that the similarity C (x, y) is greater than the variable Cxy, the following is performed. In step S1806, (x, y) is greatly inserted into the coordinate value (X, Y) as a corresponding point candidate, and the variable Cx
cxy is substituted for y2, and the similarity C is substituted for the variable Cxy.
After substituting (x, y), the process proceeds to step S1807.

On the other hand, in step S1805,
If it is determined that the similarity C (x, y) is not larger than the variable Cxy, the similarity C (x, y) is determined in step S1809.
It is determined whether or not y) is greater than the variable Cxy2. If the similarity C (x, y) is greater than the variable Cxy2, the similarity C (x, y) is substituted for the variable Cxy2 in step S1810, and then the process proceeds to step S1810. The process proceeds to S1807. In step S1809, the similarity C (x, y) is set to the variable Cx
If it is determined that it is not larger than y2, the process skips step S1810 and proceeds to step S1807.

In this step S1807, it is determined whether or not processing has been performed for all points in the searched image.
If processing has been performed for all points, the next step S1
In 808, the coordinate values (X, Y) of the corresponding point candidates held at that time are represented by the maximum similarity Cxy and the second similarity Cxy.
After outputting 2, the processing operation ends. If it is determined in step S1807 that all points have not been processed, the process returns to step S1803 to determine the similarity for the next point.

As described in detail above, the pixel number calculation unit 111 operates.

If the calculation is the first time and the template size is larger, the judgment unit 108 executes an operation of judging from the obtained similarity whether or not the center point of the template is a point in an area where the distance changes greatly. I do. If the template size is smaller, an operation for determining whether or not the result is erroneous extraction is performed.

The determination unit 108 determines whether or not the center point of the template is a point in a region where the distance changes greatly by comparing the similarity Cxy with a certain threshold value Cth. This is because when the template 191 is created around the point P in FIG. 19A and the corresponding point is obtained in FIG. 19B, for example, the similarity C at the point on the line L
When (x, y) is plotted, the result is as shown in FIG. Since the point P is not a point in the region where the distance changes greatly, a peak having a high similarity C (x, y) appears as shown in FIG.

However, when the template 192 is created around the point Q in FIG. 19A and the corresponding point is obtained in FIG. 19B, for example, the similarity C (x,
When y) is plotted, it becomes as shown in FIG. This is because, as shown in the problem of the prior art, there is a region having a greatly different distance in the template.
, The similarity C (x,
The peak on the left is where y) is increased, and the peak on the right is where the similarity C (x, y) is increased due to the region S. As described above, the similarity of the corresponding point candidate of the point existing in the area where the distance greatly changes is considerably smaller than the similarity of the point in the other area. Due to this fact,
By comparing the magnitude of the similarity with a certain threshold value Cth, it is possible to determine whether or not the center point of the template is a point in a region where the distance greatly changes.

Further, whether or not the result when the template size is small is erroneous extraction is determined by using the absolute value of the difference between the maximum similarity Cxy and the second similarity Cxy2. Cx
When the difference between y and Cxy2 is very small, it is determined that a similar point was erroneously extracted because the template is small. This is because, as shown in the conventional example and the problem, when there are a plurality of similar regions such as the sky, if the template is small, there are a plurality of similarities similar to each other. To determine this, it is checked whether there are a plurality of similar similarities. Then, when it is not a point in the area where the distance greatly changes, and when it is not erroneous extraction, the determination unit 108 outputs the corresponding point extraction result. Also, when the calculation is performed for the second time, the corresponding point extraction result is output.

The template size changing unit 109 changes the template size when the determining unit 108 determines that the center point of the template is a point in the area where the distance changes greatly, and when it is determined that the template is erroneously extracted. It operates to output the new template size Sn to the data storage unit 105.

FIG. 16 is a block diagram showing the components that operate as described above in detail.
With this configuration, it is possible to select a template according to the region in the image, to perform highly accurate corresponding point extraction processing, and to obtain a highly accurate distance distribution of the subject. In addition, the template selection processing time can be reduced.

In the present embodiment, there are two types of template sizes, large and small. However, when the background is still large even if the template size is small, the template can be further reduced, or the template size can be increased. May be used.

Further, in the present embodiment, equation (3) shown in the conventional example is used to determine the similarity. However, the present invention is not limited to this, and equations (1) and (2) are used. Well,
Alternatively, a new similarity may be used.

In the present embodiment, a template matching method using a plurality of template sizes and a judging method for selecting the template size are shown.

(Seventh Embodiment) Next, a seventh embodiment of the present invention will be described.
The embodiment will be described with reference to FIGS. 21 and 22.

FIG. 21 is a block diagram showing a configuration of a corresponding point extracting apparatus according to a seventh embodiment of the present invention. In FIG. 21, the same parts as those in FIG. The code is attached. In FIG. 21, FIG.
16 is different from the configuration in FIG. 16 in that the data storage unit 105 is deleted and an edge image memory 112 and a template size determination unit 113 are added.

In FIG. 21, similarly to the sixth embodiment, an image is input from the image input means 101, and the input image is stored in the reference image memory 102 and the searched image memory 103, respectively. Address generator 1
From 04, an address indicating one point in the reference image memory 102 and the edge image memory 112 is input to the edge image memory 112. The edge image memory 112 stores the image obtained from the reference image memory 102 as an edge image using a Sobel filter or the like. And
An edge template having a determined size around the point is created and sent to the template size determining unit 113.

The template size determination section 113 determines whether or not there is a strong edge in the obtained edge template. If there is a strong edge, the template size is reduced. If there is no strong edge, the template size is increased. Is sent to the template image creation unit 106. Then, the template image creating unit 106 creates a template image of the template size around the point indicated by the address using the reference image memory 102.

In the pixel number calculation unit 111, the expression (3) described in the prior art is applied between the template image input from the template image creation unit 106 and the searched image input from the searched image memory 103. , The number of pixels of a difference between each pixel in the template and one pixel on the searched image in the entire template image that is equal to or less than a threshold ε is calculated, and the corresponding point candidate and the similarity between the points are output. .

If the calculation is performed for the first time, the determination unit 108 determines whether the template has an appropriate size. If not, a signal is sent to the template size change unit 109, and the template size is determined based on the signal. Part 1
In step 13, a new template size is determined, and the new template size is
The similarity is calculated again by the pixel number calculation unit 111 using the new template created there, and the corresponding point is determined. If the calculation is for the second time, the corresponding point extraction result is output.

The above configuration differs from the sixth embodiment in the edge image memory 112 and the template size determination unit 113.

Hereinafter, the template size determining section 113 will be described with reference to the flowchart of FIG.

In step S2201 of FIG. 22, an edge template image is input, and in the next step S2202, it is determined whether a strong edge exists in the input edge template image. The judgment is whether there is an edge stronger than a certain threshold. If a strong edge exists, the size of the template is reduced in the next step S2203, and the process proceeds to step S2204. If no strong edge exists in step S2202, the template size is increased in step S2205, and the process proceeds to step S2204. In step S2204, after outputting the template size processed in step S2203 or step S2205, the processing operation ends.

As described above, according to the present embodiment, a strong edge often exists in a region where the distance greatly changes, and thus a strong edge exists in the edge obtained using a Sobel filter or the like. When an edge exists, the corresponding point extraction processing efficiency can be improved by reducing the size of the template used.

Further, even when the template size is selected incorrectly, the template size can be changed thereafter, so that the accuracy does not deteriorate due to the selection error, and conversely, the template size suitable for each area can be changed. Since a template can be used, the accuracy of the corresponding point extraction processing can be improved.

(Eighth Embodiment) Next, an eighth embodiment of the present invention will be described.
The embodiment will be described with reference to FIG. FIG. 23 is a block diagram showing a configuration of a corresponding point extracting apparatus according to the eighth embodiment of the present invention. In FIG. 23, the same parts as those in FIG. I have.
23 differs from FIG. 16 in that the address generator 1
04 is connected to the data storage unit 15.

In FIG. 23, an image is input from image input means 101 having two cameras as shown in FIG. 24 (a), and the input image is stored in a reference image memory 102.
And stored in the image memory 103 to be searched. From the address generation unit 104, the reference image memory 10
2 is stored in the data storage unit 10.
5, the template size is sent to the template image creation unit 106.
In, a template image of the template size is created around the point indicated by the address using the reference image memory. The pixel number calculation unit 111 converts the template image input from the template image creation unit 106 with the searched image input from the searched image memory 103 based on the expression (3) described in the related art. The difference between each pixel in the template and one pixel on the searched image in the entire image is calculated as the number of pixels equal to or smaller than a threshold ε, and the corresponding point candidates and their similarities are output.

If the calculation is performed for the first time, the determination unit 108 determines whether the template has a suitable size. If not, a signal is sent to the template size changing unit 109, and the template size is determined based on the signal. The template size is changed by the change unit 109, and the changed new template size is sent to the template image creation unit 106, and the similarity is calculated again by the pixel count calculation unit 111 using the new template created there, and the corresponding Determine points. If the calculation is for the second time, the corresponding point extraction result is output.

The difference between FIG. 23 and FIG. 16 of the sixth embodiment is the operation of the data storage unit 105, which will be described below.

[0165] One template size is stored in the data storage unit 105, and when the determination unit 108 determines that the template size is not suitable, the size is changed. Then, the changed template size becomes effective in the next case. In other words, the template size uses the size used by the point immediately before that point.

As described above, based on the fact that the area of the distance edge where the template size should be reduced is continuous,
By making the template size the same as that of the previous point, the template size becomes a suitable size with a very simple algorithm, and the corresponding point extraction processing efficiency can be improved. In addition, even when the template size is incorrect, the template size can be changed thereafter, so that the accuracy is not deteriorated, and conversely, a template having a size suitable for each region can be used. The corresponding point extraction processing accuracy can be improved.

[0167]

As described in detail above, claims 1 to 5 of the present invention are described below.
According to the corresponding point extracting method of the third aspect and the corresponding point extracting apparatus of claims 11 to 13, there is an effect that a highly accurate corresponding point extracting process can be performed over the entire area without discrimination between a subject and a background in an image to be used. .

According to the corresponding point extracting method of the fourth aspect and the corresponding point extracting apparatus of the fourteenth aspect of the present invention, it is possible to reduce the number of erroneous correspondences in the whole area of an image to be used and to improve the processing speed. Play.

According to the corresponding point extracting method of the present invention and the corresponding point extracting apparatus of the present invention, the processing accuracy is improved in a region where the distance in an image to be used greatly changes, and the corresponding region is extracted. If the number is small, the processing speed is improved by simultaneously obtaining the evaluation values for two or more types of templates.

According to the corresponding point extracting method of the present invention and the corresponding point extracting apparatus of the sixteenth aspect of the present invention, the processing accuracy is improved in a region where the distance in the image to be used greatly changes, and the corresponding region is extracted. If the number is large, the processing speed is improved by sequentially obtaining the evaluation values using two or more types of templates.

According to the corresponding point extracting method and the corresponding point extracting apparatus of the seventh and eighth aspects of the present invention,
There is an effect that the processing accuracy is improved in a region where the distance in the image to be used greatly changes.

Further, according to the corresponding point extracting method of the ninth aspect of the present invention and the corresponding point extracting apparatus of the nineteenth aspect, the processing accuracy is improved in an area where the distance in the image to be used greatly changes, and the occlusion area is improved. This has the effect of improving the accuracy of the determination.

Further, according to the corresponding point extracting method of the present invention and the corresponding point extracting apparatus of the twentieth aspect, there is an effect that the processing accuracy is improved in a region where the distance in an image to be used greatly changes. .

Further, according to the corresponding point extracting method of the present invention and the corresponding point extracting apparatus of the present invention, it is possible to use a template corresponding to a partial feature of an image to be used. Thus, there is an effect that highly accurate corresponding point extraction processing can be performed over the entire area of an image.

According to the corresponding point extracting method of claims 23 to 25 and the corresponding point extracting apparatus of claims 30 to 32 of the present invention, highly accurate corresponding point extraction is performed over the entire area without discrimination between a subject and a background in an image. There is an effect that processing can be performed.

[0176] Further, claims 26, 27, and 35 of the present invention.
37. Corresponding point extracting method of claim 37, and claim 33, 34, 38-4.
According to the corresponding point extraction apparatus of 0, a template of a suitable size can be used from the beginning, and the processing speed can be improved, and the template size can be changed thereafter. Is achieved.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a corresponding point extraction processing device according to a first embodiment of the present invention.

FIG. 2 is a flowchart illustrating a processing operation of an evaluation function calculation unit in the apparatus.

FIG. 3 is a diagram illustrating a template of an area having a constant distance and an area having a greatly changed distance in the apparatus.

FIG. 4 is a diagram showing an evaluation value of an area where the distance is constant and an evaluation value of an area where the distance greatly changes in the apparatus.

FIG. 5 is a block diagram showing a configuration of a corresponding point extraction processing device according to a second embodiment of the present invention.

FIG. 6 is a flowchart showing a processing operation of an edge mask template creating unit in the apparatus.

FIG. 7 is a diagram illustrating an edge mask template creation unit in the same apparatus.

FIG. 8 is a flowchart showing a processing operation of an evaluation function calculation unit in a corresponding point extraction processing device according to a third embodiment of the present invention.

FIG. 9 is a block diagram illustrating a configuration of a corresponding point extraction processing device according to a fourth embodiment of the present invention.

FIG. 10 is a flowchart showing a processing operation of an evaluation function calculation unit in the apparatus.

FIG. 11 is a diagram showing a template in the same device.

FIG. 12 is a flowchart illustrating a processing operation of an evaluation function calculation unit in a corresponding point extraction processing device according to a fifth embodiment of the present invention.

FIG. 13 is a diagram showing a table used by a determination unit in the apparatus.

FIG. 14 is a diagram illustrating erroneous extraction in the same device.

FIG. 15 is a diagram illustrating a template in a region where the distance greatly changes in the apparatus.

FIG. 16 is a block diagram showing a configuration of a corresponding point extraction processing device according to a sixth embodiment of the present invention.

FIG. 17 is a flowchart showing a processing operation of a data storage unit in the same device.

FIG. 18 is a flowchart showing a processing operation of a pixel number calculation unit in the same device.

FIG. 19 is a diagram illustrating a template of an area having a constant distance and an area having a greatly changed distance in the apparatus.

FIG. 20 is a diagram showing a similarity between a region having a constant distance and a region where the distance greatly changes in the apparatus.

FIG. 21 is a block diagram illustrating a configuration of a corresponding point extraction processing device according to a seventh embodiment of the present invention.

FIG. 22 is a flowchart showing a processing operation of a template size determination unit in the same device.

FIG. 23 is a block diagram showing a configuration of a corresponding point extraction processing device according to an eighth embodiment of the present invention.

FIG. 24 is a diagram illustrating an example of a general image input unit.

FIG. 25 is a diagram showing a general template matching method.

FIG. 26 is a diagram illustrating erroneous extraction in a conventional corresponding point extraction processing device.

[Explanation of symbols]

 Reference Signs List 101 Image input means 102 Reference image memory 103 Image memory to be searched 104 Address generation unit 105 Data storage unit 106 Template image creation unit 107 Evaluation function operation unit 108 Judgment unit 109 Template size change unit 110 Edge mask template creation unit 111 Pixel number operation Unit 112 edge image memory 113 template size determination unit 31 template 32 template 151 template 152 template 153 area divided by edge 191 template 192 template

──────────────────────────────────────────────────続 き Continuation of the front page (51) Int.Cl. ⁶ Identification code Agency reference number FI Technical display location // G01B 11/00 G06F 15/62 415 H04N 1/40 Z

Claims (40)

[Claims] 1. A corresponding point extraction method for determining correspondence between a plurality of images based on a template matching method, wherein the template matching method uses two or more types of templates at points satisfying a specific condition. A corresponding point extraction method characterized by determining. 2. The corresponding point extraction method according to claim 1, wherein the two or more types of templates are templates in which the area of the template is changed. 3. The corresponding point extraction method according to claim 1, wherein the two or more types of templates include a template obtained by deforming a template by an edge in the template. 4. The corresponding point extracting method according to claim 1, wherein the calculation using the second and subsequent templates is performed on points that are candidates in the calculation using the previous template. 5. The method according to claim 1, further comprising: judging the operation result of the previous template based on the operation result of the first and subsequent templates so as to use the second template, and then performing the operation of the next template. Item 1. The corresponding point extraction method according to Item 1. 6. The corresponding point extraction method according to claim 1, wherein two or more kinds of templates are simultaneously operated, and one operation result is selected from among them. 7. The corresponding point extraction method according to claim 1, wherein the point that satisfies the certain condition is a point in a region where the distribution of parallax between a plurality of images changes sharply. 8. The corresponding point according to claim 7, wherein the determination as to whether or not the point is in a region where the distribution of parallax between the plurality of images changes sharply is performed using an evaluation value of template matching. Extraction method. 9. A determination as to whether or not a point in an area where the distribution of parallax between a plurality of images changes sharply is performed based on an evaluation value of template matching, a reference point, and a corresponding point of the point as candidates. 8. The corresponding point extracting method according to claim 7, wherein the method is performed using a result of the determination as to whether or not they match. 10. A determination as to whether or not a point in an area where the distribution of parallax between a plurality of images changes steeply is performed at a point in an area where the distribution of parallax between the plurality of images changes steeply. And using the number of pixels equal to or less than a certain threshold value between the pixel value of each element in the template and the pixel value of the pixel in the searched image in the similarity evaluation function. The method for extracting corresponding points according to claim 7, characterized in that: 11. A corresponding point extracting apparatus for clarifying a correspondence between a plurality of images based on a template matching method, wherein the template matching method uses two or more types of templates at points satisfying a specific condition. A corresponding point extracting apparatus characterized in that: 12. The apparatus according to claim 11, wherein the two or more types of templates are templates in which the area of the template is changed. 13. The corresponding point extracting apparatus according to claim 11, wherein the two or more types of templates include a template obtained by deforming a template by an edge in the template. 14. The method according to claim 1, wherein the calculation using the second and subsequent templates is performed on a candidate point in the calculation using the previous template.
2. The corresponding point extracting device according to 1. 15. The method according to claim 1, further comprising: judging the operation result of the previous template so as to use the second template based on the operation result of the first and subsequent templates, and then performing the operation of the next template. Item 13. The corresponding point extraction device according to Item 11. 16. The corresponding point extracting apparatus according to claim 11, wherein calculations using two or more types of templates are performed at the same time, and one calculation result is selected from the calculated results. 17. The point satisfying the specific condition is as follows.
The corresponding point extracting apparatus according to claim 11, wherein the parallax distribution between the plurality of images is a point in an area where the distribution changes sharply. 18. The method according to claim 1, wherein the determination as to whether or not the point is in a region where the distribution of parallax between the plurality of images changes sharply is performed using an evaluation value of template matching.
7. The corresponding point extraction device according to 7. 19. A determination as to whether or not a point in an area where the disparity distribution of a plurality of images changes sharply is performed based on an evaluation value of template matching and a reference point and a corresponding point of the point as candidates. 18. The corresponding point extraction device according to claim 17, wherein the determination is performed using a result of the determination as to whether or not they match. 20. A method for determining whether or not a point in an area where the distribution of parallax between a plurality of images sharply changes is determined by using a previous template for a point in an area where the distribution of parallax between the plurality of images sharply changes. And using the number of pixels equal to or less than a certain threshold value between the pixel value of each element in the template and the pixel value of the pixel in the searched image in the similarity evaluation function. 18. The corresponding point extracting device according to claim 17, wherein: 21. A template is first created from one of a plurality of input images based on a predetermined criterion, and a pixel value of each pixel of the created template and the input other The similarity is obtained using the pixel values of the pixels in the image, and a specific condition is determined from the obtained similarity,
A corresponding point extraction method, wherein another template is created based on the determination result. 22. The method according to claim 21, wherein the template is created by changing the area of the template.
The corresponding point extraction method described. 23. The corresponding point extracting method according to claim 21, wherein the judgment of a certain specific condition is performed by comparing the inputted similarity with a certain threshold value. 24. The corresponding point extracting method according to claim 21, wherein the determination of the specific condition is performed by determining when a difference between the plurality of input similarities is smaller than a certain threshold value. 25. The corresponding point extraction method according to claim 21, wherein the determination of the specific condition is performed by determining a pixel in an area where the distribution of parallax between a plurality of images changes sharply. 26. The corresponding point extraction method according to claim 21, wherein the predetermined criterion for creating a first template is a template size of a point of a pixel near a center point of the template. 27. The corresponding point extracting method according to claim 21, wherein the predetermined criterion for creating the first template is the presence or absence of a strong edge in the template. 28. A first template creating means for first creating a template from one of a plurality of input images based on the predetermined criterion, and a template created by the template creating means. A similarity calculating means for calculating a similarity by using a pixel value of each pixel and a pixel value of a pixel in the input other image; and a specification for determining a specific condition from the similarity calculated by the similarity calculating means. A corresponding point extracting apparatus, comprising: condition determining means; and second template creating means for creating another template based on a result of the determination by the specific condition determining means. 29. The second template creating means,
29. The corresponding point extracting apparatus according to claim 28, wherein the template is created by changing the area of the template. 30. The corresponding point extracting apparatus according to claim 28, wherein said specific condition judging means judges a specific condition by comparing the input similarity with a certain threshold value. 31. The specific condition judging means judges a specific condition by judging when a difference between a plurality of inputted similarities is smaller than a certain threshold value.
8. The corresponding point extraction device according to 8. 32. The corresponding point according to claim 28, wherein said specific condition determining means determines a specific condition by determining pixels in an area where the distribution of parallax between a plurality of images changes sharply. Extraction device. 33. The corresponding point extracting apparatus according to claim 28, wherein the predetermined criterion for creating a first template is a template size of a point of a pixel near a center point of the template. 34. The corresponding point extracting apparatus according to claim 28, wherein the predetermined criterion for creating the first template is the presence or absence of a strong edge in the template. 35. A template is created from one of a plurality of input images using a template size used at a point obtained immediately before, and a pixel value of each pixel of the created template and A similarity is calculated using the pixel values of the pixels in the other input image, and when the input similarity is smaller than a certain threshold is determined from the calculated similarity, a template is determined based on the determination result. A corresponding point extracting method, characterized in that a template having a changed area is created. 36. A template is created from one of a plurality of input images using a template size used at a point obtained immediately before, and a pixel value of each pixel of the created template and The similarity is obtained using the pixel values of the pixels in the other input image, and when the difference between the plurality of input similarities from the obtained similarity is smaller than a certain threshold value, it is determined. A corresponding point extraction method, characterized in that a template in which the area of the template is changed based on the result is created. 37. A template is created using a template size used at a point obtained immediately before from one image of a plurality of input images, and a pixel value of each pixel of the created template is calculated. A similarity is obtained by using the pixel values of the pixels in the input other image, and a pixel in an area where a distribution of parallax between a plurality of images changes sharply from the obtained similarity is determined. A corresponding point extraction method, characterized in that a template in which the area of the template is changed based on the template is created. 38. A first template creation means for creating a template from one of a plurality of input images using a template size used at a point obtained immediately before, and the first template Similarity calculating means for calculating a similarity using the pixel value of each pixel of the template created by the creating means and the pixel value of the pixel in the input other image; and a similarity calculated by the similarity calculating means. Determining means for determining when the degree of similarity input from is smaller than a certain threshold value, and generating a template in which the area of the template is changed based on the determination result of the determining means.
A corresponding point extraction device, comprising: 39. A first template creating means for creating a template using a template size used at a point obtained immediately before from one image of a plurality of input images, and said first template Similarity calculating means for calculating a similarity using the pixel value of each pixel of the template created by the creating means and the pixel value of the pixel in the input other image; and a similarity calculated by the similarity calculating means. Determining means for determining when a difference between a plurality of similarities input from a plurality is smaller than a certain threshold value, and a second template generating means for generating a template in which the area of the template is changed based on the determination result of the determining means And a corresponding point extracting device comprising: 40. A first template creating means for creating a template using a template size used at a point obtained immediately before from one of a plurality of input images, and said first template Similarity calculating means for calculating a similarity using the pixel value of each pixel of the template created by the creating means and the pixel value of the pixel in the input other image; and a similarity calculated by the similarity calculating means. A determining means for determining a pixel in an area in which the distribution of parallax between a plurality of images changes sharply from a plurality of images; A corresponding point extracting apparatus, comprising: