JPH0991436A - Image processing method and device therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To recognize the accurate 3-dimensional shape of an object by restoring a 1st frame shape and a 2nd frame shape corresponding to an edge segment of a reference image thereby integrating the 1st and 2nd frame shapes. SOLUTION: A corresponding point is executed between a reference image and a 1st image, and depth information of each segment extracted from the reference image is calculated (S206). Then a 3-dimensional shape (1st frame shape) of a segment is restored based on the distance information resulting from the 2-dimensional shape of the segment of the reference image. Then the image input (S300), edge extraction (S302), segmentation and extraction of 2-dimensional shape (S304), retrieval of corresponding point with the reference image (S306), and restoration of 3-dimensional shape of segment (S308) are executed, restoration processing to the 3-dimensional shape (2nd frame shape) is executed for the segment and part of the 2-dimensional shape of the reference image is restored.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method and apparatus for capturing an object from a plurality of positions and processing each of the obtained images to grasp the shape of the object.

[0002]

2. Description of the Related Art Conventionally, as a method of obtaining the shape and position of an object in a three-dimensional space, for example, two image pickup devices are used to image the object from the right and left directions, and the obtained left gray image and right gray image are obtained. The principle of triangulation is performed by correlating the left and right images based on the similarity in brightness between the images (hereinafter, these two images are collectively referred to as “left and right images”) or shape feature information such as edges. There is known a method for obtaining the depth distance of each point or each area, that is, a so-called stereoscopic stereoscopic method.

Here, in the correspondence between right and left images, so-called corresponding point search, correspondence candidate points are generally determined by using an epipolar line (projective straight line) constraint obtained from the geometrical arrangement of an image pickup device, for example, a CCD camera. By doing so, the search efficiency is improved. Furthermore, it is attempted to change the window size of the correlation, calculate the degree of correspondence in consideration of edge connectivity, or use the epipolar constraint obtained from three or more imaging devices to further eliminate the ambiguity of corresponding points. The method of doing, etc. is being tried.

In particular, when it is desired to obtain not only distance information of each area of a target object or edge points but also three-dimensional shape information, a segment such as a line segment, a quadratic curve or the like is extracted from features such as edges. A method is widely known in which the three-dimensional position of each segment is obtained by triangulation, by extracting the correspondence between the left and right segments.

However, from a set of left and right images,
Frequently, correspondence cannot be obtained between the left and right segments due to occlusion, noise, segment breaks, and the like, and the shape is horizontal with respect to the epipolar so that it cannot be restored as a three-dimensional shape. Therefore, a method has been proposed in which stereoscopic vision is performed between images obtained from a plurality of viewpoints and the three-dimensional shapes obtained at each viewpoint are integrated to restore a more complete three-dimensional shape. .
Here, the integration of the three-dimensional shapes obtained from the respective viewpoints is realized by estimating a plane from an occlusion area formed by each segment, particularly each line segment, and determining the plane information (for example, Japanese Laid-Open Patent Publication 63-244174, JP-A-3-2680
No. 69 publication).

[0006]

The conventional method for picking up an image of the target object described above and obtaining the three-dimensional shape and position of the object from the image information has the following problems.

1. Unlike a simple shape such as a building block, for a general object having a more complicated shape (for example, an industrial part), the appearance of edges between the left and right images, the connection state of segments, In many cases, the shapes of the segments do not match, and it is difficult to obtain correct correspondence information by the method of finding correspondence between the segments.

2. In the conventional method, a plane is estimated from an occlusion area formed by each line segment, and segments obtained from each viewpoint are integrated using the surface information. Generally, each segment is an occlusion area. Is not always configured, and rather, an occlusion region is often not obtained, and thus such a method includes many ambiguities in estimation, and as a result, segment integration by the above method is difficult.

3. Further, even though the segments obtained from the respective viewpoints are actually the segments obtained from the same object, due to the reasons described in 1. and the error of the camera calibration, the length, shape type, etc. , It is general that the positions do not match, and it is difficult to judge and integrate the identities of the segments obtained from each viewpoint.

The present invention has been made to solve the above problems, and an object thereof is to provide an image processing apparatus,
To recognize the exact three-dimensional shape of an object. Especially,
Even if features (for example, segment position, shape, length, etc.) do not match between the left and right images, correct correspondence is obtained, and the target shape, for example, three-dimensional shape or It is to be able to obtain a two-dimensional shape. In addition, in order to restore a more complete shape from the target object, images are taken from different viewpoints, and in the process of integrating the features from those viewpoints, accurate determination is performed without making ambiguity such as by plane estimation. To recognize the shape of an object. In addition, features obtained from each viewpoint (eg, segment length, shape, position, etc.)
Is to enable reliable integration of features and reconstruction of shapes, even if their identity is compromised or if they do not form an occlusion area or plane.

[0011]

The present invention is an image processing method for recognizing the shape of an object using a reference image, a first image and a second image obtained by imaging the object from at least three different positions. In, the following processing is performed. First, using the correspondence between the reference image and the edge segment extracted from the first image, the frame shape of the object is calculated as the first frame shape from the edge segment in the reference image. Secondly, the frame shape of the object is calculated as the second frame shape from the edge segment in the reference image by using the correspondence between the reference image and the edge segment extracted from the second image. Third, by restoring the first frame shape corresponding to the edge segment of the reference image, and then restoring the second frame shape corresponding to the edge segment portion of the reference image that was not restored in the first frame shape. , The first frame shape and the second frame shape are integrated.

In another method invention, when the second frame shape is calculated, the second frame shape is calculated from the edge segment of the reference image which is not obtained as the first frame shape, and the calculation is performed as such. The frame shape of the entire object is obtained by a simple addition of the first frame shape and the second frame shape.

In the apparatus invention, the edge segment and the point corresponding to each edge segment of the reference image and the first image are obtained, and the frame shape of the object is determined from the edge segment in the reference image to the first frame based on the correspondence information. The first shape calculation means for calculating the shape and the corresponding edge segment and point between each edge segment of the reference image and the second image are obtained, and the frame shape of the object is calculated from the edge segment in the reference image based on this correspondence information. Second shape calculation means for calculating as the second frame shape, and corresponding to the edge segment portion of the reference image which is not restored by the first frame shape after restoring the first frame shape corresponding to the edge segment of the reference image Integration that integrates the first frame shape and the second frame shape by restoring the second frame shape Characterized by providing a stage.

In another apparatus invention, the second shape calculation means calculates the second frame shape from the edge segment of the reference image which is not obtained as the first frame shape, and the integrating means calculates the first frame. The frame shape of the entire object is obtained by simply adding the shape and the second frame shape.

In the above apparatus, reference image input means for obtaining a reference image to extract edge segments, and first image input means for obtaining a first image and extracting edge segments,
Second image input means for obtaining the second image and extracting the edge segment may be added. Further, the present apparatus may be an apparatus that processes an image obtained off-line, or an apparatus that executes the imaging of an object and the recognition processing of the shape of the object as a series of processes. The image processing method and the apparatus thereof according to the present invention need only image the object from at least three different positions, and may image the object from four or more different positions.

[0016]

In the image processing method for obtaining the shape and position of an object in a three-dimensional space from the imaged image of the object, in order to obtain a more complete shape of the object, images taken from a plurality of viewpoints are used. Process. At that time,
As shown in FIG. 4, a reference image, for example, a left image in binocular stereoscopic vision is set and its viewpoint is fixed, while an image paired with the reference image, for example, the right image is moved while moving the viewpoint. Obtain images from multiple viewpoints so that many shapes can be restored.

Thus, by setting one reference image with a fixed viewpoint, moving the viewpoint of the other image paired with the reference image, and always performing corresponding point search and shape restoration based on the reference image, The shape of the target object that is as complete as possible, for example, a three-dimensional shape or a two-dimensional shape, is obtained from an image without making an integrated determination by complicated processing.

For example, when binocular stereoscopic vision is performed based on edges, the edges are extracted from the left and right images. Then get the line segment and the quadratic segment from the edge,
The edge of the left image and the edge of the right image are associated with each other. At this time, the corresponding points are obtained with high reliability by using the segment information, and the three-dimensional frame shape of the object is obtained from the corresponding point information and the segment information.

At this time, as shown in FIG. 1, first, the correspondence between the segments and the corresponding points between the reference image and the first image are obtained, and the information is used to extract 3 segments from the segments in the reference image.
Find the first frame shape of the dimension. Similarly, the correspondence relationship and corresponding point of the segment between the reference image and the second image are obtained, and the three-dimensional second frame shape is obtained from the segment in the reference image using the information. Similarly, for other images, the third frame shape, the fourth frame shape, and the like are obtained.

Next, the plurality of frame shapes obtained as described above are integrated. That is, the three-dimensional shape of the object is restored by directly adopting the first frame shape corresponding to the edge segment of the basic image. Next, in the edge segment of the basic image, only the edge segment that has not been restored to the three-dimensional shape is extracted from the second frame shape to restore the three-dimensional shape of the object. , Only the edge segment that has not been restored to the 3D shape,
The operation of extracting from the frame shape and restoring the three-dimensional shape of the object is repeatedly executed. By such an integrating operation, the final three-dimensional shape of the object is restored.

Further, as shown in FIG.
When determining each frame shape, the edge segment of the basic image for which the frame shape has already been calculated may not be calculated as the frame shape. In this case, when the frame shapes are integrated to obtain the final three-dimensional shape of the object, the integration operation may be a simple addition of the frame shapes.

According to the above method, the restoration process to the three-dimensional shape is performed with the segment of the reference image as a reference, and the segment that has already been restored to the three-dimensional shape is further restored in shape. Is not processed. That is, in the relationship between the reference image and the image captured from a certain viewpoint, the corresponding segment in the reference image is restored to a three-dimensional shape. The subsequent processing, that is, the reference image and the image captured from another viewpoint. When obtaining the correspondence of the segment with the generated image, the segment already adopted as the three-dimensional shape is excluded. Therefore, there is no need to calculate an unnecessary correspondence relationship, and there is no plurality of segment candidates corresponding to one element of the three-dimensional shape. Therefore, a determination process for integrating these segment candidates is performed. You don't even have to do it. Therefore, even though the segments actually correspond to elements having different three-dimensional shapes, it is determined that those segments have a correspondence relationship, or conversely, even though there is a correspondence relationship, the correspondence relationship is It is possible to solve a problem that often occurs in the conventional image processing method such as determining that there is no. Moreover, since the above-mentioned useless calculation is not performed, the processing is simplified and the processing speed is improved.

Further, according to the present invention, basically, since the corresponding points are searched on the edge basis, the three-dimensional shape can be reliably obtained even if the occlusion property of the segments and the left-right matching are not maintained. Furthermore, by using a plurality of viewpoints, the trinocular effect can be used as a result. For example, when obtaining a corresponding point between the reference image and the i-th image, images from other viewpoints ( (J-th image) is used for verification of correspondence candidate selection, or the search area of the i-th image is set using the corresponding point information with the j-th image that has already been obtained with the reference image. It is possible to efficiently and reliably find corresponding points and restore the shape. Here, an example of restoring a three-dimensional shape consisting of line segments and ellipses based on edges is shown, but according to this method, other shapes (points, planes, etc.)
It can also be applied to three-dimensional reconstruction of areas. Furthermore, although the restoration of the three-dimensional shape is shown here, the application to the restoration of the two-dimensional shape is easy.

[0024]

BEST MODE FOR CARRYING OUT THE INVENTION The present invention will be described below based on specific embodiments. FIG. 2 is a block diagram showing the configuration of the image processing apparatus according to the embodiment. N cameras C1 to Cn are provided to image an object (not shown) from different n points. The cameras C1 to Cn respectively output the image signals output from the cameras C1 to Cn,
It is connected to image memories (frame grabbers) F1 to Fn for storing. The image data stored in each of the image memories F1 to Fn can be read by the computer 5 via the data bus 3. The computer 5 processes the input image data and recognizes the three-dimensional shape of the object.
The images captured by the cameras C1 to Cn are the monitor M1.
Each can be displayed in ~ Mn.

FIG. 1 shows three objects to be executed by the apparatus of this embodiment.
It is an explanatory view showing restoration processing of a three-dimensional shape. The camera arranged at the reference position is referred to as a reference camera, the image captured by the reference camera is referred to as a reference image, the i-th camera arranged at a viewpoint different from that of the reference camera is referred to as an i-th camera, and an image from the i-th camera is called. It is called the i-th image. The three-dimensional shape restored using the reference image and the i-th image is called the i-th frame shape. Furthermore, a three-dimensional shape obtained by integrating all (n-1) frame shapes is called a final three-dimensional shape. still,
In FIG. 1, five cameras (n = 5) from the reference camera and the first camera to the fourth camera are used, and the obtained images and frame shapes are the first image to the fourth image and the first frame. 4 from the shape to the 4th frame shape (n-1)
The image and shape of

FIG. 3 is a flow chart showing the processing procedure of the computer 5. Hereinafter, the processing procedure for obtaining the final three-dimensional shape shown in FIG. 2 will be described with reference to FIG. In step 100, the reference camera is instructed to capture an image and the reference image is input to the image memory F1. Next, in step 102, the edges of the image are extracted by applying an appropriate edge extraction operator to the reference image. Then, in step 104, the edge is segmented and a two-dimensional shape extraction is performed on each segment. For example, a straight line and a quadratic curve are approximated to a certain segment to extract a line segment, an elliptic arc, an arc, and the like.

On the other hand, an image pickup command is similarly given to the first camera, and the same processing as the reference image is executed. That is, each processing of image input (step 200), edge extraction (step 202), segmentation and extraction of a two-dimensional shape of the segment (step 204) is performed.

Then, in step 206, a corresponding point search is executed between the reference image and the first image, and the depth distance information of each segment extracted from the reference image is calculated. Next, in step 208, the three-dimensional shape of the segment is restored from the distance information and the two-dimensional shape of the segment in the reference image obtained in step 104, that is, the first frame shape is obtained and stored in the memory. It is registered in the three-dimensional shape table.

At this time, in the actual image, a part of the reference image is affected by noise or occlusion.
In many cases, only the three-dimensional shape is restored to the three-dimensional shape. Therefore, in order to more completely reconstruct the three-dimensional shape from the reference image, the second image obtained from a different viewpoint from the first image is next obtained.
A three-dimensional shape restoration process is executed using the image and the reference image.

Similar to the processing of the first image, image input (step 300), edge extraction (step 302), segmentation and two-dimensional shape extraction (step 304) are performed, and corresponding point search with the reference image is performed. (Step 306),
The three-dimensional shape of the segment is restored (step 308). This three-dimensional shape restoration processing is executed only for the segments in the reference image that have not yet been restored to the three-dimensional shape. That is, in the processing of the second image, the segment that cannot be restored by the restoration processing of the first image in step 208 is executed. Then, the segment for which the three-dimensional shape restoration process has been executed is additionally registered in the three-dimensional shape table as the second frame shape. As a result, a part of the two-dimensional shape of the reference image that has not been restored between the reference image and the first image is restored.

The same processing as described above is executed for the third and fourth images obtained by the third and fourth cameras,
The third frame shape and the fourth frame shape are required. At this time, similarly, it is executed only for the segment which has not been restored to the three-dimensional shape in the reference image. It should be noted that even if more n cameras are used, the same processing is repeatedly executed, and the object
The dimensional shape is restored.

As described above, in the process of integrating three-dimensional shapes obtained from different viewpoints in order to restore a more complete shape from an image, the two-dimensional shape segment of the reference image is always referred to, The ambiguity in the integrated judgment can be removed, and further, since the judgment processing is not required, the processing speed can be increased. Furthermore, by using a plurality of viewpoints, corresponding points can be accurately and efficiently obtained by, for example, three-eye view. For example, when obtaining the corresponding point between the reference image and the i-th image, an image from another viewpoint (j-th image) is used for verification of the correspondence candidate selection, or
For example, by setting the search area of the i-th image using the corresponding point information with the j-th image that has been obtained with the reference image,
It is possible to speed up the processing by limiting the search area and improve the reliability by removing the incorrect correspondence.

[Brief description of drawings]

FIG. 1 is an explanatory diagram showing an image processing method according to the present invention.

FIG. 2 is a block diagram showing an image processing apparatus according to a specific embodiment of the present invention.

FIG. 3 is a flowchart showing a processing procedure of the apparatus of the embodiment.

FIG. 4 is an explanatory view showing another method of image processing according to the present invention.

[Explanation of symbols]

 5 ... Computer C1-Cn ... Camera F1-Fn ... Image memory M1-Mn ... Monitor

Claims (5)

[Claims] 1. An image processing method for recognizing the shape of an object using a reference image, a first image, and a second image obtained by imaging an object from at least three different positions, the reference image and the first image The frame shape of the object is calculated as the first frame shape from the edge segment in the reference image using the correspondence between the edge segments extracted from the image and the edge segment extracted from the reference image and the second image. The frame shape of the object is calculated as a second frame shape from the edge segment in the reference image by using the correspondence relationship of 1), and the first frame shape is restored corresponding to the edge segment of the reference image, and then the first frame shape is restored. By restoring the second frame shape corresponding to the edge segment portion of the reference image which has not been restored in the frame shape, the first frame is restored. An image processing method characterized by integrating the over arm shape and a second frame shape. 2. An image processing method for recognizing the shape of an object using a reference image, a first image and a second image obtained by picking up an object from at least three different positions. The frame shape of the object is calculated as the first frame shape from the edge segment in the reference image using the correspondence between the edge segments extracted from the image and the edge segment extracted from the reference image and the second image. The frame shape of the object is calculated as the second frame shape from the edge segment that does not correspond to the first frame shape among the edge segments in the reference image, using the correspondence relationship between the first frame shape and the second frame shape. An image processing method characterized by integrating a shape and a frame shape of an object. 3. An image processing apparatus for recognizing a shape of an object from images obtained by picking up an image of an object from a plurality of positions, a first image different from the reference image obtained by picking up the object from a reference position, and the reference position. In the edge segment extracted from the first image obtained by imaging the object from the position, the corresponding edge segment and point between the edge segments of each image are obtained, and the edge segment in the reference image based on the correspondence information. A first shape calculation means for calculating the frame shape of the object as a first frame shape; a reference image obtained by imaging the object from a reference position; and a second image different from the reference position and the first position. In the edge segment extracted from the second image obtained by imaging the object from the position, between the edge segments of each image Second shape calculation means for obtaining the corresponding edge segment and point and calculating the frame shape of the object as the second frame shape from the edge segment in the reference image based on the correspondence information, and corresponding to the edge segment of the reference image. After restoring the first frame shape, the first frame shape and the second frame shape are restored by restoring the second frame shape corresponding to the edge segment portion of the reference image that was not restored in the first frame shape. An image processing apparatus comprising: an integration unit for integrating. 4. An image processing apparatus for recognizing a shape of an object from images obtained by picking up an image of an object from a plurality of positions, a first image different from the reference image obtained by picking up the object from a reference position, and the reference position. In the edge segment extracted from the first image obtained by imaging the object from the position, the corresponding edge segment and point between the edge segments of each image are obtained, and the edge segment in the reference image based on the correspondence information. A first shape calculation means for calculating the frame shape of the object as a first frame shape; a reference image obtained by imaging the object from a reference position; and a second image different from the reference position and the first position. Of the edge segments extracted from the second image obtained by imaging the object from the position, edges that do not correspond to the first frame shape A second shape calculating means for calculating the frame shape of the object as a second frame shape from the segment, and an integrating means for integrating the first frame shape and the second frame shape into the frame shape of the object. A characteristic image processing device. 5. The image processing apparatus includes a reference image input means for extracting an edge segment from a reference image obtained by capturing the object from a reference position, and capturing the object from a first position different from the reference position. First image input means for extracting an edge segment from a first image obtained by the above, and an edge segment from a second image obtained by imaging the object from a second position different from the reference position and the first position. The image processing apparatus according to claim 3 or 4, further comprising a second image input unit for extracting.