JPH0944677A - Correspondent point extracting device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract correspondent points between plural images picked up from plural positions with high accuracy regardlessly of the state of the surface of a subject. SOLUTION: The correspondent points of images A and B provided by picking up the image of a subject 11 at two positions while moving a camera 12 are found by a correspondent point extracting part 15. Based on the correlative value of images A and B provided at such a time, a difficult subject judging part 16 judges whether the subject is difficult to extract the correspondent points such as the surface of the subject 11 is flat or has repeated patterns or not, and when it is difficult one, the pattern suitable for correspondent point extraction is projected onto the subject 11 by a projector 13. While using the images A and B provided by picking up the image of the subject 11, on which the pattern is projected, by the camera 12, the correspondent point extraction part 15 extracts the correspondent points corresponding to the projected pattern.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a corresponding point extracting apparatus for finding corresponding points between a plurality of images of the same subject taken from different viewpoints.

[0002]

2. Description of the Related Art Conventionally, a distance map to an object is obtained by extracting corresponding points between a plurality of images obtained by placing a camera on a plurality of different viewpoints of the same object, and based on this distance map. There is known a technique for obtaining a three-dimensional shape of a subject. A template matching method has been conventionally known as a representative method of the corresponding point extracting method for clarifying the correspondence between the plurality of images. This template matching method determines a corresponding point by considering a template that surrounds one point in the reference image to be matched and calculating the similarity between the template and the area on the searched image. It is a method to do.

Next, the template matching method will be conceptually described with reference to FIG. In FIG. 10, it is assumed that one point P on the right ear of the person in the reference image 100 corresponds to the searched image 101. First, the template 102 of a certain size centered on the point P
Create Next, the template 102 is moved in the searched image 101, the similarity is calculated at each point in the searched image 101, and the point having the highest similarity is associated with the point P of the reference image 100. It is a point.

The calculation of the degree of similarity includes a method of using the pixel value difference as in the following equation (1) and a method of using the pixel value correlation value as in the equation (2).

[0005]

[Equation 1]

In the above equations (1) and (2), F (i,
j) represents the image 101 to be searched, and A (i,
j) represents the template 102. That is, each of the above equations indicates the degree of similarity when the position of the template 102 is moved by (x, y). In addition, when calculated by the above formula (1), the corresponding point is the point where E (x, y) is the minimum, and the theoretical minimum value of E (x, y) is 0. Further, when calculated by the above formula (2), the corresponding point is the point where σ (x, y) is maximum, and theoretically the maximum value of σ (x, y) is 1.

Next, FIG. 11 shows a conventional system for obtaining a distance map of a subject from a plurality of images. In the figure,
110 is a subject, 111 is a camera, and 112 is a computer. The corresponding points are obtained by the template matching method described above using a plurality of images captured by moving the camera 111 to a plurality of positions, and the distance map of the subject 110 is obtained based on the principle of triangulation. These calculations are performed by the computer 112. The three-dimensional shape of the subject 110 can be obtained from the obtained distance map.

[0008]

However, the above-described template matching method used for extracting corresponding points has a low accuracy of extracting corresponding points on the entire surface of an object having a uniform surface without a pattern. Similarly, the corresponding point extraction accuracy of points on the image with low contrast is also low. Also,
The corresponding point extraction accuracy of points on the surface having a repetitive pattern also becomes low. As described above, the template matching method has a problem that some subjects are not good at finding corresponding points, and it is not possible to extract corresponding points with high accuracy for any subject.

Conventionally, there have been active spot extraction methods for corresponding points, such as a spot light projection method for projecting Sbot light onto a subject and a slit light projection method for projecting slit light. Requires a set of images to measure one point and requires a long processing time. The slit light projection method also requires N times of image input processing to obtain N slit images, which requires a long processing time. As described above, the conventional active method has a problem that it is necessary to project light a plurality of times to obtain a distance map of one subject, and the processing time is correspondingly long.

Therefore, an object of the present invention is to make it possible to perform highly accurate extraction of corresponding points when picking up an image of any subject. In addition, the number of times of actively projecting light is reduced.

[0011]

According to the present invention, corresponding point extracting means for extracting corresponding points of a plurality of images obtained by imaging the same subject from a plurality of different viewpoints, and the subject performing the corresponding point extraction. A weak subject discriminating means for discriminating whether or not the subject is weak, and a pattern projecting means for projecting a predetermined pattern onto the subject when the weak subject discriminating means discriminates that the subject is weak Is provided.

[0012]

According to the present invention, the pattern projecting means projects the projection pattern on the subject when the subject determining means determines that the subject is a poor subject. The corresponding point extracting means can use the image of the subject on which the pattern is projected and extract the corresponding points based on the pattern.

[0013]

1 shows the configuration of a system for obtaining a distance map of an object according to a first embodiment of the present invention. In the figure, 11 is a subject, 12 is a camera that images the subject 11, 13 is a projection device that projects light onto the subject 11, 14 is a camera movement mechanism that moves the camera 12, 1
Reference numeral 5 is a corresponding point extraction unit that obtains corresponding points between a plurality of images obtained from the camera 12, and reference numeral 16 is a weak subject determination unit. Note that the corresponding point extraction unit 15 and the weak subject determination unit 16 may be configured by one computer 17.

FIG. 2 is a flow chart showing the operation of obtaining the distance map of the object 11 by the system configuration of FIG.

Next, the process of obtaining the distance map of the object 11 will be described with reference to FIGS. First, in the image A input step S1, the subject 11 is imaged by the camera 12 at a certain position, and the image A is input to the corresponding point extraction unit 15. Next, in the camera moving step S2, the camera moving mechanism 1
4, the camera 12 is moved to another position. At this time, the movement amount and position of the camera 12 can be measured by using the camera moving mechanism 14. Also, when the camera is moved by hand, the movement amount and position of the camera 12 can be measured by a gyro or the like.

Next, in the image B input step S3, the subject 11 is imaged again by the camera 12 at the position after the movement, and the image B is input to the corresponding point extracting unit 15. With respect to the two images A and B obtained in the above steps S2 and S3, the corresponding point extracting unit 15 obtains corresponding points between the images A and B in a corresponding point extracting step S4. In the corresponding point extraction processing step S4, the corresponding points are obtained by the template matching method, but not only the corresponding points are output but also the correlation value obtained by the template matching calculation is output.

Next, in a weak subject discriminating step S5, it is discriminated based on the correlation value obtained in the corresponding point extraction processing step S4 whether or not the subject 11 in the image is a poor subject by a method described later. When it is determined that the subject 11 is not a weak subject, a distance map of the subject 11 is obtained based on the principle of triangulation using the corresponding points and the camera position.

When the subject 11 is determined to be a poor subject, the projection device 13 projects a predetermined pattern onto the subject 11 in the projection step S6. By this projection, a pattern that allows template matching to the subject 11 is obtained, so that it is possible to extract corresponding points with high accuracy without depending on the state of the surface of the object, and to obtain a highly accurate distance map.

Subsequently, the camera 12 is moved to a plurality of different positions, and the subject 11 on which the pattern is projected at each position.
Is imaged, and corresponding points are obtained from the obtained images by the template matching method. Then, based on the triangulation, the distance map of the subject 11 is obtained using the corresponding points and the information on the position of the camera 12.

When the corresponding point extraction processing is performed using the image of the subject 11 on which the pattern is projected, the corresponding points can be obtained from the pattern. You can ask.
However, a method is also conceivable in which a plurality of patterns are prepared, the subject determination process which is not good even in a situation where the patterns are projected is performed, and if the patterns are bad, the patterns are exchanged.

The pattern used here is not a laser spot light but a pattern that allows template matching only by one projection and has a clear contrast. By using such a pattern, it is possible to avoid a problem such as slit light in which projection must be performed many times to improve accuracy. FIG. 3 shows an example of a pattern. This is a pattern in which the center of the pattern is brightest and the brightness values are concentrically lowered toward the periphery.

Next, a method of discriminating an unfavorable subject will be described. FIG. 4 shows an example of distribution of correlation values obtained by the template matching method used for extracting corresponding points. The distribution appears two-dimensional, but here it is one-dimensional for easy viewing. FIG. 4A shows the case of using a general subject,
A sharp peak appears at the corresponding point. FIG. 4B shows a distribution when a subject having a small contrast such as a white wall without a pattern is used, and there are a plurality of fine peaks. Figure 4 (c)
Is a distribution when a subject having a repeating pattern is used, and a plurality of somewhat strong peaks appear. Thus, in the case of FIGS. 4B and 4C, ambiguity remains in the detection of corresponding points.

Therefore, in the present embodiment, the subject that is not good at is discriminated based on the distribution of the correlation value. That is, when the distribution of the correlation values is other than that shown in FIG. 4A, it is determined that the subject is a poor subject.

As described above, according to the present embodiment, since a highly accurate corresponding point cannot be conventionally obtained, a low-contrast flat object or a repetitive pattern in which a highly accurate distance map cannot be obtained. Even for existing subjects, corresponding points can be extracted for any subject by discriminating based on the distribution of correlation values and projecting a pattern that can be template-matched. Can be obtained from the image pair.

FIG. 5 shows the second embodiment, and FIG. 6 shows the operation flow. In the first embodiment, the determination of the poor subject is made based on the distribution of the correlation values obtained by the corresponding point extraction processing unit 15. However, in the second embodiment, the distance map of the subject 11 is obtained. The configuration and processing described in FIG. 5 and FIG. 6 are performed so that the subject that is not good at the discrimination is determined before the corresponding points are extracted.

In FIGS. 1, 2 and 5 and 6, the processing of the poor subject discriminating unit 16 (step S5) and the corresponding point extracting unit 1 are performed.
Since only the processing of step 5 (step S4) is replaced, only this part will be described. The weakness discrimination according to the second embodiment is performed by using the frequency component of the image. Specifically, a mask process such as a Sobel filter is performed on the obtained plurality of images, and then Fourier transform is performed. When this is displayed logarithmically, in the case of a general image with contrast, a distribution showing very minute changes in each frequency range is obtained as shown in FIG. 7A. Although the image used is two-dimensional, it is shown in one dimension for easy understanding. However, if the subject has no pattern, the DC component becomes extremely small when Fourier transform is applied to the image after filtering for edge extraction as shown in FIG. 7B. Further, if it has a periodic pattern, a plurality of large peaks of the amplitude spectrum appear as shown in FIG. 7 (c).

Therefore, in the present embodiment, the subject which is not good at is discriminated by utilizing the distribution as seen on the frequency axis. Note that, as a result of this determination, the subsequent processing is the same as that of the first embodiment, and therefore its explanation is omitted.

According to the second embodiment, not the distribution of the correlation value as in the first embodiment but the frequency component of the image is used to discriminate the weak subject, so that the corresponding point extraction processing is performed. It is possible to discriminate the subject which is not good before the. As a result, the corresponding point extraction process is performed once for any subject, and the processing time can be shortened.

FIG. 8 shows a third embodiment, in which a pattern selection section 18 is added to the first embodiment of FIG. In FIG. 8, the weak subject determination unit 16 determines that the subject is a poor subject, the subject 11 determines what type of poor subject, and the pattern selection unit 18 selects a pattern to be projected according to the type. A plurality of patterns are prepared in the projection device 13, and the pattern selected by the pattern selection unit 18 is projected onto the subject 11. Other processes are the same as in the first embodiment. It goes without saying that the pattern selection unit 18 may also be used in the second embodiment shown in FIG.

Next, a method of discriminating the type of subject that is not good will be described. FIG. 4B for the first embodiment
FIGS. 7B and 7C relating to FIG. 7C and the second embodiment.
As described above, the distribution of correlation values and the distribution of edges are greatly different depending on the subject. Therefore, focusing on the difference in this distribution, the type of subject that is not good is determined based on this distribution.

Next, a pattern suitable for the type of subject which is not good will be described. First, for a subject with no pattern and low contrast, the contrast is emphasized and a pattern with contrast is projected. For example, the pattern is as shown in FIG. Further, for a subject having a repeating pattern, a pattern completely different from the repeating pattern is emphasized rather than the contrast, and such a pattern is projected. For example, a pattern as shown in FIG. 9A is projected. Further, if it is determined from the correlation distribution or the edge distribution that a horizontal stripe pattern is formed, a concentric pattern centered on the image center is projected as shown in FIG. 9B.

By projecting in this way, a subject on which a pattern suitable for a subject that is not good is projected is imaged,
Corresponding points can be extracted with high accuracy, and a distance map with high accuracy can be obtained. Since the subsequent processing is the same as in the first and second embodiments, the description thereof will be omitted.

According to the present embodiment, whether or not the subject for which the distance map is to be obtained is a poor subject, and if it is a poor subject, it is also determined what kind of poor subject, and a pattern suitable for the type is set as the subject. By projecting, the corresponding points can be extracted with high accuracy, and a distance map with high accuracy can be obtained.

[0034]

As described above, according to the present invention,
In the corresponding point extracting apparatus that obtains corresponding points between a plurality of images, it is possible to extract corresponding points with high accuracy even for a subject that is difficult to obtain with high accuracy.
Further, highly accurate corresponding point extraction can be performed in a short processing time only by projecting the pattern a few times. In addition, it is possible to save electric power as compared with the case of projecting each time by discriminating the subject that is not good and projecting the pattern only on the subject that is poor.

Further, by discriminating the type of the subject that is not good at, it is possible to project a pattern suitable for the subject,
It is possible to extract corresponding points with higher accuracy.

[Brief description of drawings]

FIG. 1 is a configuration diagram showing a first embodiment of the present invention.

FIG. 2 is a flowchart showing an operation of the first embodiment.

FIG. 3 is a configuration diagram showing an example of a pattern projected on a subject used in the first embodiment.

FIG. 4 is a characteristic diagram showing a distribution example of correlation values obtained by a template matching method.

FIG. 5 is a configuration diagram showing a second embodiment of the present invention.

FIG. 6 is a flowchart showing an operation of the second embodiment.

FIG. 7 is a characteristic diagram showing an example of a frequency spectrum of an image.

FIG. 8 is a configuration diagram showing a third embodiment of the present invention.

FIG. 9 is a configuration diagram showing an example of a pattern projected on a subject.

FIG. 10 is a configuration diagram for explaining template matching.

FIG. 11 is a configuration diagram of a conventional system for obtaining a distance map.

[Explanation of symbols]

 11 Subject 12 Camera 12 Projection Device 15 Corresponding Point Extraction Unit 16 Poor Subject Discrimination Unit 18 Pattern Selection Unit

 ─────────────────────────────────────────────────── ─── Continued Front Page (72) Inventor Motohiro Ishikawa 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc. (72) Inventor Kotaro Yano 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Incorporated (72) Inventor Katsumi Iijima 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc. (72) Inventor Nao Kurahashi 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc.

Claims (4)

[Claims] 1. A corresponding point extracting means for extracting corresponding points of a plurality of images obtained by picking up the same subject from a plurality of different viewpoints, and whether or not the subject is a subject who is not good at extracting the corresponding points. A corresponding point extraction device comprising: a subject which is not good at determining whether or not, and a pattern projecting means which projects a predetermined pattern onto the subject when the subject is weak at the subject. 2. The discrimination by the subject discriminating means which is not good at
The corresponding point extracting device according to claim 1, wherein the corresponding point extracting device is configured to carry out based on correlation values of the plurality of images obtained from the corresponding point extracting means. 3. The discrimination by the subject discriminating means, which is not good,
The corresponding point extracting device according to claim 1, wherein the corresponding point extracting device is performed based on a frequency component of an input image. 4. The pattern projecting means is configured to selectively project a plurality of patterns, and the weak subject discriminating means is characterized in that the correlation value obtained from the corresponding point extracting means or the frequency of the input image. The correspondence according to claim 1, further comprising: a pattern selecting unit configured to determine a type of the subject which is not good on the basis of the component, and which selects a pattern projected by the pattern projecting unit according to the determined type of the poor subject. Point extractor.